Sep 17 15:53:33.822111 ip-10-0-143-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:53:33.822123 ip-10-0-143-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:53:33.822130 ip-10-0-143-187 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:53:33.822283 ip-10-0-143-187 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:53:43.904563 ip-10-0-143-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:53:43.904582 ip-10-0-143-187 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4386f097610f45499de82faad9e0e8b3 --
Sep 17 15:55:47.701912 ip-10-0-143-187 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:55:51.225467 ip-10-0-143-187 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:51.225467 ip-10-0-143-187 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:55:51.225467 ip-10-0-143-187 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:51.225467 ip-10-0-143-187 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:55:51.225467 ip-10-0-143-187 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:55:51.226243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.225532    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:55:51.231080 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231057    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.231080 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231077    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.231080 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231083    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.231080 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231087    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231092    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231097    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231101    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231107    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231111    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231115    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231120    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231123    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231125    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231128    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231131    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231133    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231136    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231139    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231141    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231144    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231146    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231149    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231152    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.231249 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231155    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231157    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231163    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231166    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231169    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231172    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231174    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231177    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231179    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231182    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231185    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231187    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231190    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231192    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231196    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231198    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231201    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231204    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231208    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.231758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231213    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231216    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231219    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231222    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231224    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231227    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231230    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231233    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231235    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231238    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231241    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231246    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231249    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231252    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231255    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231258    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231261    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231263    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231266    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.232227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231269    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231272    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231274    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231277    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231279    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231282    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231285    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231287    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231291    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231293    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231296    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231299    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231302    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231304    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231307    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231310    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231312    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231315    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231317    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231320    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.232708 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231322    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231326    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231329    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231332    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231334    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231337    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231340    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231343    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231345    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231349    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231353    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231356    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231358    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231361    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231364    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231366    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231369    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.231371    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232102    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232109    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.233240 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232114    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232117    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232120    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232123    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232125    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232128    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232130    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232133    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232136    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232139    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232142    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232145    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232147    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232150    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232152    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232155    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232158    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232161    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232163    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232166    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.233735 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232169    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232171    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232174    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232176    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232179    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232181    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232184    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232186    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232189    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232192    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232194    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232197    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232200    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232203    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232206    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232209    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232211    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232214    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232217    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232220    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232223    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.234231 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232226    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232228    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232231    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232234    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232237    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232240    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232243    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232246    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232250    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232252    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232255    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232257    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232260    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232263    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232265    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232268    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232271    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232273    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232276    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232281    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.234765 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232285    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232287    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232290    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232296    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232300    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232304    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232307    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232310    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232313    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232316    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232319    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232322    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232325    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232328    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232332    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232335    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232337    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232340    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232342    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.235271 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232345    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232348    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232351    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232353    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232356    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232358    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232361    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232364    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232367    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232370    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232372    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232375    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232377    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232380    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232382    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232385    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232391    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232777    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232783    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.235748 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232786    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232789    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232792    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232795    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232798    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232800    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232803    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232806    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232808    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232811    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232814    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232816    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232819    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232823    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232827    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232830    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232832    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232835    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232838    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.236222 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232840    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232843    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232845    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232848    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232850    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232853    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232856    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232858    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232861    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232863    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232867    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232870    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232872    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232875    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232877    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232880    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232883    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232885    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232888    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232891    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.236690 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232893    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232896    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232898    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232901    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232903    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232907    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232910    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232913    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232917    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232919    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232922    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232924    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232927    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232929    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232932    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232935    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232937    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232940    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232942    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232945    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.237235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232947    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232951    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232955    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232958    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232962    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232965    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232968    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232971    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232973    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232976    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232979    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232982    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232984    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232987    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232990    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232992    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232995    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.232998    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233001    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.237750 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233004    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233006    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233010    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233012    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233015    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233018    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233020    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233023    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233026    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233028    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233031    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233033    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233036    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233038    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233041    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233045    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233048    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233052    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233055    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233136    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:55:51.238251 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233184    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233191    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233200    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233204    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233208    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233212    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233216    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233219    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233222    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233226    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233231    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233234    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233237    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233240    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233243    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233247    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233250    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233255    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233258    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233261    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233265    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233268    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233272    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:55:51.238760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233275    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233278    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233281    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233284    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233287    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233290    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233293    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233296    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233300    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233303    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233306    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233309    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233311    2562 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233314    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233318    2562 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233321    2562 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233324    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233327    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233330    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233338    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233341    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233344    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233347    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233350    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233353    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:55:51.239341 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233356    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233359    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233364    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233367    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233370    2562 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233374    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233377    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233380    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233383    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233386    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233389    2562 flags.go:64] FLAG: --help="false"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233392    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233396    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233399    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233402    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233405    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233409    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233412    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233415    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233418    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233420    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233423    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233427    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233429    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:55:51.239981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233432    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233435    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233439    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233442    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233445    2562 flags.go:64] FLAG: --lock-file=""
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233448    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233451    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233454    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233459    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233462    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233466    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233469    2562 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233472    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233476    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233479    2562 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233481    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233485    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233489    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233493    2562 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233495    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233498    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233501    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233504    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233507    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233510    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:55:51.240575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233513    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233520    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233523    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233526    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233529    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233532    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233537    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233540    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233543    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233547    2562 flags.go:64] FLAG: --port="10250"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233550    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233553    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06ecbb3d24ec0971b"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233556    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233559    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233562    2562 flags.go:64] FLAG: --register-node="true"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233565    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233569    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233573    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233577    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233580    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233584    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233587    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233590    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233593    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233595    2562 flags.go:64] FLAG: --runonce="false"
Sep 17 15:55:51.241192 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233598    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233601    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233604    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233607    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233610    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233624    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233628    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233631    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233634    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233637    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233640    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233642    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233646    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233649    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233652    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233657    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233661    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233664    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233669    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233672    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233675    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233678    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233681    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233684    2562 flags.go:64] FLAG: --v="2"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233688    2562 flags.go:64] FLAG: --version="false"
Sep 17 15:55:51.241849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233694    2562 flags.go:64] FLAG: --vmodule=""
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233698    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.233701    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233815    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233819    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233822    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233825    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233827    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233830    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233833    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233835    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233838    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233841    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233844    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233846    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233849    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233852    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233854    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233857    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233860    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233863    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.242456 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233865    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233868    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233872    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233875    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233878    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233881    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233883    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233888    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233891    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233894    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233898    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233902    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233905    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233908    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233911    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233913    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233916    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233919    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233921    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.243003 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233924    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233927    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233929    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233932    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233934    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233937    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233940    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233942    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233945    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233947    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233950    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233953    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233956    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233958    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233961    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233965    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233968    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233970    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233973    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233976    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.243474 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233978    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233981    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233984    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233989    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.233993    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234005    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234010    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234013    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234015    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234018    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234021    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234024    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234027    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234029    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234032    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234035    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234038    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234040    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234043    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.243982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234046    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234049    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234052    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234056    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234059    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234062    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234065    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234068    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234072    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234075    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234077    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234080    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234083    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234086    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234088    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234091    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234094    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234099    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234102    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.244441 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234104    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234107    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234110    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.234112    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.234755    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.242405    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.242421    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.242438    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.242444    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242498    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242503    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242507    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242510    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242513    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.244950 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242517    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242520    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242523    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242527    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242530    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242533    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242535    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242538    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242541    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242544    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242546    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242549    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242552    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242555    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242557    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242560    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242563    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242567    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242571    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.245316 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242574    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242577    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242579    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242582    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242585    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242589    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242592    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242595    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242597    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242600    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242603    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242606    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242608    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242627    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242631    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242633    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242636    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242639    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242641    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242644    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.245788 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242647    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242649    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242652    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242655    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242657    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242660    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242662    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242665    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242668    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242671    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242673    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242676    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242679    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242681    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242684    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242687    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242690    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242693    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242696    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242699    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.246277 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242702    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242704    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242708    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242712    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242715    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242719    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242722    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242724    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242727    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242730    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242732    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242735    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242738    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242740    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242743    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242745    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242748    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242750    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242754    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.246869 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242756    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242759    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242761    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242764    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242767    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242770    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242772    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242775    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242777    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242780    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242783    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242786    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242789    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242791    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242794    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.247332 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242797    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.242802    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242915    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242920    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242923    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242925    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242928    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242931    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242933    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242936    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242939    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242941    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242944    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242947    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:55:51.247731 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242950    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242952    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242955    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242957    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242960    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242963    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242966    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242968    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242972    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242976    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242979    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242981    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242984    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242987    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242989    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242993    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.242997    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243000    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243004    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:55:51.248072 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243008    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243011    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243014    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243017    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243020    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243022    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243025    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243028    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243030    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243033    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243035    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243038    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243041    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243043    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243046    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243048    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243051    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243054    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243057    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243059    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:55:51.248529 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243062    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243065    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243068    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243070    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243073    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243075    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243078    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243081    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243084    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243086    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243089    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243092    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243094    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243097    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243100    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243102    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243105    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243107    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243110    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243112    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:55:51.249034 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243115    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243117    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243120    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243122    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243125    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243127    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243130    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243133    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243135    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243138    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243140    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243143    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243146    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243149    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243152    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243155    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243157    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243160    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243162    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243165    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:55:51.249513 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243168    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243170    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243173    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243176    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243179    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243181    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243184    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:51.243186    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.243192    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.243285    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.245468    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.246289    2562 server.go:1019] "Starting client certificate rotation"
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.246386    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:55:51.250022 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.247206    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:55:51.250339 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.249878    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:55:51.251008 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.250993    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:55:51.257021 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.257004    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:55:51.258424 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.258411    2562 log.go:25] "Validated CRI v1 image API"
Sep 17 15:55:51.259171 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.259155    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:55:51.262901 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.262882    2562 fs.go:135] Filesystem UUIDs: map[2817b3c3-8b74-4c91-978a-c5338cc55860:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ff591c93-d07b-4295-8b8b-7bb626bad574:/dev/nvme0n1p3]
Sep 17 15:55:51.262968 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.262899    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:55:51.267539 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.267432    2562 manager.go:216] Machine: {Timestamp:2026-09-17 15:55:51.266974108 +0000 UTC m=+2.122674660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099242 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26080b95b45bf70350352db9f3d89f SystemUUID:ec26080b-95b4-5bf7-0350-352db9f3d89f BootID:4386f097-610f-4549-9de8-2faad9e0e8b3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:d2:2e:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:d2:2e:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:bc:05:1e:2f:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:55:51.267539 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.267532    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:55:51.267666 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.267603    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:55:51.270253 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.270225    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:55:51.270385 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.270256    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:55:51.271111 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.271100    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:55:51.271148 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.271114    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:55:51.271148 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.271121    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:55:51.271148 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.271132    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:55:51.271229 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.271197    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:55:51.275547 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275535    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:55:51.275677 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275667    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:55:51.275724 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275716    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:55:51.275757 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275728    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:55:51.275757 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275744    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:55:51.275757 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275753    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:55:51.275840 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.275765    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:55:51.276117 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.276104    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:55:51.276167 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.276126    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:55:51.287352 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.287330    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 15:55:51.289295 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.289277    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:55:51.289748 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.289733    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:55:51.289813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.289795    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:55:51.289845 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.289831    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:55:51.291336 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291324    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291341    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291347    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291353    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291362    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291368    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291387    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:55:51.291394 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291394    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291402    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291407    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291416    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291420    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291444    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:55:51.291574 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.291450    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:55:51.295517 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.295501    2562 server.go:1268] "Started kubelet"
Sep 17 15:55:51.295569 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.295532    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:55:51.295716 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.295649    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:55:51.295774 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.295749    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:55:51.324213 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.324189    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:55:51.324282 ip-10-0-143-187 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:55:51.326729 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.326394    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:55:51.327538 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.327498    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:55:51.327853 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.327824    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:55:51.328442 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.328425    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:55:51.332389 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.331441    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-187.ec2.internal.18d626ca147cb08a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-187.ec2.internal,UID:ip-10-0-143-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-187.ec2.internal,},FirstTimestamp:2026-09-17 15:55:51.295316106 +0000 UTC m=+2.151016659,LastTimestamp:2026-09-17 15:55:51.295316106 +0000 UTC m=+2.151016659,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-187.ec2.internal,}"
Sep 17 15:55:51.333466 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.333450    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:55:51.333548 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.333471    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:55:51.333728 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.333706    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.334141 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.334124    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:55:51.334221 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.334147    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:55:51.334221 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.334124    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:55:51.334299 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.334234    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:55:51.334299 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.334239    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:55:51.335940 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.335920    2562 factory.go:153] Registering CRI-O factory
Sep 17 15:55:51.335940 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.335939    2562 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:55:51.336087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.335992    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:55:51.336087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.336001    2562 factory.go:55] Registering systemd factory
Sep 17 15:55:51.336087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.336010    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:55:51.336087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.336027    2562 factory.go:103] Registering Raw factory
Sep 17 15:55:51.336087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.336040    2562 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:55:51.336335 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.336323    2562 manager.go:349] Starting recovery of all containers
Sep 17 15:55:51.345939 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.345924    2562 manager.go:354] Recovery completed
Sep 17 15:55:51.349089 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.349072    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:55:51.349307 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.349290    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:55:51.349923 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.349909    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.352102 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352088    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.352161 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352115    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.352161 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352125    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.352549 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352537    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:55:51.352549 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352548    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:55:51.352549 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352548    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.352708 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.352561    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:55:51.353783 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353767    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.353853 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353797    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.353853 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353811    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.353954 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353912    2562 policy_none.go:49] "None policy: Start"
Sep 17 15:55:51.353954 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353924    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:55:51.353954 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.353937    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:55:51.354996 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.354984    2562 policy_none.go:47] "Start"
Sep 17 15:55:51.356589 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.356573    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c8g29"
Sep 17 15:55:51.364486 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.364424    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-187.ec2.internal.18d626ca17df30c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-187.ec2.internal,UID:ip-10-0-143-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-187.ec2.internal,},FirstTimestamp:2026-09-17 15:55:51.352103105 +0000 UTC m=+2.207803657,LastTimestamp:2026-09-17 15:55:51.352103105 +0000 UTC m=+2.207803657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-187.ec2.internal,}"
Sep 17 15:55:51.368069 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.368055    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c8g29"
Sep 17 15:55:51.369680 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.369560    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-187.ec2.internal.18d626ca17df7013  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-187.ec2.internal,UID:ip-10-0-143-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-187.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-187.ec2.internal,},FirstTimestamp:2026-09-17 15:55:51.352119315 +0000 UTC m=+2.207819866,LastTimestamp:2026-09-17 15:55:51.352119315 +0000 UTC m=+2.207819866,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-187.ec2.internal,}"
Sep 17 15:55:51.391183 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391167    2562 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.391209    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391222    2562 server.go:87] "Starting device plugin registration server"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391446    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391465    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391537    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391608    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.391630    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.392096    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:55:51.402880 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.392145    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.467678 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.467645    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:55:51.468814 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.468800    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:55:51.468912 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.468856    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:55:51.468912 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.468883    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:55:51.469059 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.468927    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:55:51.471148 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.471131    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:51.492177 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.492129    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.492977 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.492961    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.493064 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.492989    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.493064 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.492999    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.493064 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.493022    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.506232 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.506217    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.506232 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.506239    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-187.ec2.internal\": node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.520436 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.520415    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.570007 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.569982    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"]
Sep 17 15:55:51.570095 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.570041    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.571463 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.571440    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.571553 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.571469    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.571553 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.571483    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.572929 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.572917    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.573070 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.573108 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573086    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.573630 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573600    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.573697 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573644    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.573697 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573656    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.573768 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573600    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.573768 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573723    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.573768 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.573735    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.574822 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.574809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.574895 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.574839    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:51.575450 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.575435    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:51.575528 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.575462    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:51.575528 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.575481    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:51.594255 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.594235    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-187.ec2.internal\" not found" node="ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.598541 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.598523    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-187.ec2.internal\" not found" node="ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.620832 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.620812    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.636279 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.636260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.636330 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.636285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.636330 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.636300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c3c972c2ec7ee80e8e1f20047dd8543-config\") pod \"kube-apiserver-proxy-ip-10-0-143-187.ec2.internal\" (UID: \"0c3c972c2ec7ee80e8e1f20047dd8543\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.720877 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.720849    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.736859 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.736973 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c3c972c2ec7ee80e8e1f20047dd8543-config\") pod \"kube-apiserver-proxy-ip-10-0-143-187.ec2.internal\" (UID: \"0c3c972c2ec7ee80e8e1f20047dd8543\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.736973 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.736973 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.737084 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c3c972c2ec7ee80e8e1f20047dd8543-config\") pod \"kube-apiserver-proxy-ip-10-0-143-187.ec2.internal\" (UID: \"0c3c972c2ec7ee80e8e1f20047dd8543\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.737084 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.736941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9264f1e21920152ed80f06b282288df9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal\" (UID: \"9264f1e21920152ed80f06b282288df9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.821928 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.821846    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:51.898414 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.898377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.903224 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:51.903024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:51.922170 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:51.922140    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.022688 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.022651    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.123269 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.123194    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.223799 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.223762    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.246312 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.246284    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:55:52.246753 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.246412    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:55:52.324801 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.324659    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.372902 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.372866    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:50:51 +0000 UTC" deadline="2028-03-06 14:37:47.938194575 +0000 UTC"
Sep 17 15:55:52.372902 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.372897    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12862h41m55.565300974s"
Sep 17 15:55:52.425783 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.425724    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.463498 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.463477    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:52.517915 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.517890    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:55:52.519330 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.519312    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:55:52.519435 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.519346    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:55:52.519435 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.519360    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:55:52.526495 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:52.526479    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-187.ec2.internal\" not found"
Sep 17 15:55:52.554126 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:52.554091    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9264f1e21920152ed80f06b282288df9.slice/crio-32328d20d3ce837d78bb4d3bdbfb15a0102389acac3957dcee1687a82ded2860 WatchSource:0}: Error finding container 32328d20d3ce837d78bb4d3bdbfb15a0102389acac3957dcee1687a82ded2860: Status 404 returned error can't find the container with id 32328d20d3ce837d78bb4d3bdbfb15a0102389acac3957dcee1687a82ded2860
Sep 17 15:55:52.554551 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:52.554531    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3c972c2ec7ee80e8e1f20047dd8543.slice/crio-558a6a35544ac25029f8fc896672edf68c0e57c83f6697f8d4a4578cf0ea4429 WatchSource:0}: Error finding container 558a6a35544ac25029f8fc896672edf68c0e57c83f6697f8d4a4578cf0ea4429: Status 404 returned error can't find the container with id 558a6a35544ac25029f8fc896672edf68c0e57c83f6697f8d4a4578cf0ea4429
Sep 17 15:55:52.556744 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.556723    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:52.560169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.560149    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:55:52.634336 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.634313    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:52.650915 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.650896    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:55:52.651814 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.651802    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal"
Sep 17 15:55:52.661975 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.661959    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:55:52.834075 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:52.834055    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:53.114803 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.114730    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:55:53.276738 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.276695    2562 apiserver.go:52] "Watching apiserver"
Sep 17 15:55:53.288345 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.288105    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:55:53.289168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.288914    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-lphmx","openshift-network-operator/iptables-alerter-rrkdd","openshift-ovn-kubernetes/ovnkube-node-qh6xq","kube-system/global-pull-secret-syncer-xhmrn","kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal","openshift-multus/multus-additional-cni-plugins-vbxj6","openshift-multus/multus-x9rwz","openshift-multus/network-metrics-daemon-b9nng","openshift-network-diagnostics/network-check-target-6pwd9","kube-system/konnectivity-agent-vvclt","openshift-cluster-node-tuning-operator/tuned-zdfqz"]
Sep 17 15:55:53.290644 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.290606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.290759 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.290709    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:55:53.291981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.291959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.293692 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.293672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.295175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.295059    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.295175 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.295140    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:55:53.296809 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.296602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.298406 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.298383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.299551 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.299533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.301376 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.301056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.302485 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.302465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:53.302676 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.302544    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:55:53.304002 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.303983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.305428 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.305411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.312709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.312527    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:55:53.312709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.312564    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.312709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.312578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.312909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.312820    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:55:53.313087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313072    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:55:53.313274 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313258    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.313342 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313279    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:55:53.313342 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313302    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.313439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313403    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.314439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313753    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.314439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313813    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.314439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.313981    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.314439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.314146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:55:53.314439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.314306    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:55:53.315262 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.314880    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:55:53.315262 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315097    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:55:53.315262 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315143    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:55:53.315262 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.315491 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315302    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:55:53.315491 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315365    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.315491 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:53.315491 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315470    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8f7rk\""
Sep 17 15:55:53.315710 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:55:53.315710 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:55:53.315710 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315677    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wfxkg\""
Sep 17 15:55:53.315855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315712    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k9w7s\""
Sep 17 15:55:53.315855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r8m68\""
Sep 17 15:55:53.315855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315816    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9m8hk\""
Sep 17 15:55:53.316007 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fz5j4\""
Sep 17 15:55:53.316007 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.315932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhnkc\""
Sep 17 15:55:53.316099 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.316024    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-76pft\""
Sep 17 15:55:53.316099 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.316073    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:55:53.316193 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.316155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:55:53.334092 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.334070    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:55:53.335959 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.335942    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:55:53.345471 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-etc-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.345563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-env-overrides\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.345563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.345563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-system-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-kubernetes\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-var-lib-kubelet\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87ffe04-0550-4937-9c43-f290c275f12f-iptables-alerter-script\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx7w\" (UniqueName: \"kubernetes.io/projected/a87ffe04-0550-4937-9c43-f290c275f12f-kube-api-access-fsx7w\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-log-socket\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.345733 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-config\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-tuned\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9b5c\" (UniqueName: \"kubernetes.io/projected/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-kube-api-access-l9b5c\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-k8s-cni-cncf-io\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-kubelet\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdwk\" (UniqueName: \"kubernetes.io/projected/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-kube-api-access-dxdwk\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-system-cni-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/694737e1-5cd2-4739-8254-49c167544e79-agent-certs\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-modprobe-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.346014 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.345989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-var-lib-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-kubelet\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-systemd-units\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-netns\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-systemd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-dbus\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-registration-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-device-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-multus-certs\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-etc-kubernetes\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz87\" (UniqueName: \"kubernetes.io/projected/f948d78e-92e1-42d9-bc1a-be5e30b8c899-kube-api-access-xmz87\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysconfig\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-run\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346449    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095422d7-51d2-4c32-817f-f4ec7024face-host\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.346476 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-host\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxvm\" (UniqueName: \"kubernetes.io/projected/b2fc0529-7ca7-4545-829d-033b33052d9f-kube-api-access-gqxvm\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87ffe04-0550-4937-9c43-f290c275f12f-host-slash\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-socket-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-socket-dir-parent\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-multus\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.347289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.346741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095422d7-51d2-4c32-817f-f4ec7024face-serviceca\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.348560 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.348661 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-kubelet-config\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.348730 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.348784 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-lib-modules\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.348784 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-ovn\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.348880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-slash\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.348880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.348981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cnibin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.348981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-netns\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.348981 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.348925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-hostroot\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.352863 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.352839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-netd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.352950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.352894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovn-node-metrics-cert\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.352950 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.352929    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-conf\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.353051 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.352959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-sys\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.353051 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.352991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-bin\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.353051 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-script-lib\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.353200 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-daemon-config\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.353200 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.353200 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7n4\" (UniqueName: \"kubernetes.io/projected/f01d3d75-6ebd-4acc-b209-50035e01489c-kube-api-access-jb7n4\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.353380 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/694737e1-5cd2-4739-8254-49c167544e79-konnectivity-ca\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.353380 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-tmp\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.353380 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-node-log\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.353380 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5vr\" (UniqueName: \"kubernetes.io/projected/43070168-32cf-48c0-8a1f-1211c0ee47dd-kube-api-access-hb5vr\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.353563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cni-binary-copy\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.353563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-bin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.353563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.354028 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.353604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-sys-fs\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.354103 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-conf-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.354103 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.354208 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:53.354208 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-cnibin\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.354208 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.354208 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tnf\" (UniqueName: \"kubernetes.io/projected/095422d7-51d2-4c32-817f-f4ec7024face-kube-api-access-26tnf\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.354404 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.354220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-systemd\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.379973 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.379947    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:55:53.403301 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.403276    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-79v2p"
Sep 17 15:55:53.419666 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.419641    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-79v2p"
Sep 17 15:55:53.455234 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.455343 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/694737e1-5cd2-4739-8254-49c167544e79-agent-certs\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.455343 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-modprobe-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.455343 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-var-lib-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455343 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-kubelet\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-systemd-units\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-netns\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-systemd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-dbus\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-kubelet\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-registration-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-var-lib-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-modprobe-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-device-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-netns\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-systemd-units\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.455507 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-multus-certs\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-device-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-etc-kubernetes\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455570    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-etc-kubernetes\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-registration-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz87\" (UniqueName: \"kubernetes.io/projected/f948d78e-92e1-42d9-bc1a-be5e30b8c899-kube-api-access-xmz87\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-multus-certs\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-dbus\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455691    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysconfig\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysconfig\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-systemd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-run\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095422d7-51d2-4c32-817f-f4ec7024face-host\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-host\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-run\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.456175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-d\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455888    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/095422d7-51d2-4c32-817f-f4ec7024face-host\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxvm\" (UniqueName: \"kubernetes.io/projected/b2fc0529-7ca7-4545-829d-033b33052d9f-kube-api-access-gqxvm\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-host\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87ffe04-0550-4937-9c43-f290c275f12f-host-slash\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-socket-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.455997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-socket-dir-parent\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87ffe04-0550-4937-9c43-f290c275f12f-host-slash\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-multus\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-socket-dir-parent\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095422d7-51d2-4c32-817f-f4ec7024face-serviceca\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-multus\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-socket-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-kubelet-config\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-lib-modules\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-ovn\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-slash\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-ovn\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cnibin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/249026f0-6f4d-43a2-96ae-bd00ac176d62-kubelet-config\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-netns\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-slash\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-netns\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-lib-modules\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cnibin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-run-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.457909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-hostroot\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-hostroot\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-netd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovn-node-metrics-cert\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-conf\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-sys\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-bin\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-script-lib\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-daemon-config\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/095422d7-51d2-4c32-817f-f4ec7024face-serviceca\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7n4\" (UniqueName: \"kubernetes.io/projected/f01d3d75-6ebd-4acc-b209-50035e01489c-kube-api-access-jb7n4\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-sysctl-conf\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/694737e1-5cd2-4739-8254-49c167544e79-konnectivity-ca\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.456735    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-sys\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.458721 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.456800    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:53.956775391 +0000 UTC m=+4.812475944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.456951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-netd\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-host-cni-bin\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-tmp\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-node-log\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5vr\" (UniqueName: \"kubernetes.io/projected/43070168-32cf-48c0-8a1f-1211c0ee47dd-kube-api-access-hb5vr\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cni-binary-copy\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-bin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/694737e1-5cd2-4739-8254-49c167544e79-konnectivity-ca\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-sys-fs\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-conf-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-script-lib\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-cnibin\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-node-log\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.459415 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-daemon-config\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tnf\" (UniqueName: \"kubernetes.io/projected/095422d7-51d2-4c32-817f-f4ec7024face-kube-api-access-26tnf\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-systemd\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-conf-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-etc-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-cni-bin\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-env-overrides\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43070168-32cf-48c0-8a1f-1211c0ee47dd-sys-fs\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-cni-binary-copy\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-env-overrides\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-cni-binary-copy\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.457995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-etc-openvswitch\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-system-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-multus-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-kubernetes\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460076 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-var-lib-kubelet\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-system-cni-dir\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87ffe04-0550-4937-9c43-f290c275f12f-iptables-alerter-script\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx7w\" (UniqueName: \"kubernetes.io/projected/a87ffe04-0550-4937-9c43-f290c275f12f-kube-api-access-fsx7w\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-log-socket\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-config\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f01d3d75-6ebd-4acc-b209-50035e01489c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-kubernetes\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-tuned\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-systemd\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.458126    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-log-socket\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458230    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-cnibin\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.458256    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:53.95823358 +0000 UTC m=+4.813934118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9b5c\" (UniqueName: \"kubernetes.io/projected/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-kube-api-access-l9b5c\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-k8s-cni-cncf-io\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-kubelet\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.460650 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdwk\" (UniqueName: \"kubernetes.io/projected/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-kube-api-access-dxdwk\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-system-cni-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f01d3d75-6ebd-4acc-b209-50035e01489c-system-cni-dir\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-var-lib-kubelet\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovnkube-config\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-host-run-k8s-cni-cncf-io\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87ffe04-0550-4937-9c43-f290c275f12f-iptables-alerter-script\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.458888    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2fc0529-7ca7-4545-829d-033b33052d9f-var-lib-kubelet\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.459578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-ovn-node-metrics-cert\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.459597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-tmp\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.459743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/694737e1-5cd2-4739-8254-49c167544e79-agent-certs\") pod \"konnectivity-agent-vvclt\" (UID: \"694737e1-5cd2-4739-8254-49c167544e79\") " pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.461211 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.460842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2fc0529-7ca7-4545-829d-033b33052d9f-etc-tuned\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.474884 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.474805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal" event={"ID":"9264f1e21920152ed80f06b282288df9","Type":"ContainerStarted","Data":"32328d20d3ce837d78bb4d3bdbfb15a0102389acac3957dcee1687a82ded2860"}
Sep 17 15:55:53.476058 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.476038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal" event={"ID":"0c3c972c2ec7ee80e8e1f20047dd8543","Type":"ContainerStarted","Data":"558a6a35544ac25029f8fc896672edf68c0e57c83f6697f8d4a4578cf0ea4429"}
Sep 17 15:55:53.480041 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.480004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz87\" (UniqueName: \"kubernetes.io/projected/f948d78e-92e1-42d9-bc1a-be5e30b8c899-kube-api-access-xmz87\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.492462 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.492435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7n4\" (UniqueName: \"kubernetes.io/projected/f01d3d75-6ebd-4acc-b209-50035e01489c-kube-api-access-jb7n4\") pod \"multus-additional-cni-plugins-vbxj6\" (UID: \"f01d3d75-6ebd-4acc-b209-50035e01489c\") " pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.494201 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.494181    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:53.494305 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.494209    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:53.494305 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.494222    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:53.494305 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.494278    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:53.994259956 +0000 UTC m=+4.849960512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:53.494478 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.494443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5vr\" (UniqueName: \"kubernetes.io/projected/43070168-32cf-48c0-8a1f-1211c0ee47dd-kube-api-access-hb5vr\") pod \"aws-ebs-csi-driver-node-wj797\" (UID: \"43070168-32cf-48c0-8a1f-1211c0ee47dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.494887 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.494843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxvm\" (UniqueName: \"kubernetes.io/projected/b2fc0529-7ca7-4545-829d-033b33052d9f-kube-api-access-gqxvm\") pod \"tuned-zdfqz\" (UID: \"b2fc0529-7ca7-4545-829d-033b33052d9f\") " pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.496784 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.496766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx7w\" (UniqueName: \"kubernetes.io/projected/a87ffe04-0550-4937-9c43-f290c275f12f-kube-api-access-fsx7w\") pod \"iptables-alerter-rrkdd\" (UID: \"a87ffe04-0550-4937-9c43-f290c275f12f\") " pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.496784 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.496783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tnf\" (UniqueName: \"kubernetes.io/projected/095422d7-51d2-4c32-817f-f4ec7024face-kube-api-access-26tnf\") pod \"node-ca-lphmx\" (UID: \"095422d7-51d2-4c32-817f-f4ec7024face\") " pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.498957 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.498936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdwk\" (UniqueName: \"kubernetes.io/projected/15584b8e-f9c4-4f04-89b6-7ea3e4527a7c-kube-api-access-dxdwk\") pod \"multus-x9rwz\" (UID: \"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c\") " pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.505606 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.505587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9b5c\" (UniqueName: \"kubernetes.io/projected/9202fcd5-a26c-424e-a4c6-5031f21ec9bb-kube-api-access-l9b5c\") pod \"ovnkube-node-qh6xq\" (UID: \"9202fcd5-a26c-424e-a4c6-5031f21ec9bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.605270 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.605240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrkdd"
Sep 17 15:55:53.612592 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.612567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:55:53.631787 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.631735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdfqz"
Sep 17 15:55:53.638521 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.638500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797"
Sep 17 15:55:53.645160 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.645138    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vbxj6"
Sep 17 15:55:53.656689 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.656672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x9rwz"
Sep 17 15:55:53.663865 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.663848    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lphmx"
Sep 17 15:55:53.668530 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.668510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:55:53.962039 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.961956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:53.962039 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:53.962018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:53.962252 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.962129    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:53.962252 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.962202    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:54.962188649 +0000 UTC m=+5.817889203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:53.962252 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.962130    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:53.962389 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:53.962281    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:54.962266598 +0000 UTC m=+5.817967150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:54.063354 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.063324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:54.063485 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.063452    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:54.063485 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.063466    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:54.063485 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.063475    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:54.063593 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.063523    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:55.063510134 +0000 UTC m=+5.919210690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:54.389982 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.389951    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694737e1_5cd2_4739_8254_49c167544e79.slice/crio-15d778628038fda74e9062f1a5b4202cd168952ffeb16956931f89d7bcc2c989 WatchSource:0}: Error finding container 15d778628038fda74e9062f1a5b4202cd168952ffeb16956931f89d7bcc2c989: Status 404 returned error can't find the container with id 15d778628038fda74e9062f1a5b4202cd168952ffeb16956931f89d7bcc2c989
Sep 17 15:55:54.391418 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.391374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fc0529_7ca7_4545_829d_033b33052d9f.slice/crio-58fd66ccbc9bb7e78da18f53b30a2e9ff6afe94b9319be986415f4d074e3ba88 WatchSource:0}: Error finding container 58fd66ccbc9bb7e78da18f53b30a2e9ff6afe94b9319be986415f4d074e3ba88: Status 404 returned error can't find the container with id 58fd66ccbc9bb7e78da18f53b30a2e9ff6afe94b9319be986415f4d074e3ba88
Sep 17 15:55:54.392396 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.392374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15584b8e_f9c4_4f04_89b6_7ea3e4527a7c.slice/crio-706ce49316fb185212faa30c83e4779872562cc7d6952f608180a07282865f8b WatchSource:0}: Error finding container 706ce49316fb185212faa30c83e4779872562cc7d6952f608180a07282865f8b: Status 404 returned error can't find the container with id 706ce49316fb185212faa30c83e4779872562cc7d6952f608180a07282865f8b
Sep 17 15:55:54.394253 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.393891    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095422d7_51d2_4c32_817f_f4ec7024face.slice/crio-0a7535ffd5a3974ac53573d7bb4bdc29598da4e512e58201b785515766db4ae1 WatchSource:0}: Error finding container 0a7535ffd5a3974ac53573d7bb4bdc29598da4e512e58201b785515766db4ae1: Status 404 returned error can't find the container with id 0a7535ffd5a3974ac53573d7bb4bdc29598da4e512e58201b785515766db4ae1
Sep 17 15:55:54.395566 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.395513    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87ffe04_0550_4937_9c43_f290c275f12f.slice/crio-74c4eb12667d3bc8e90849892a290a5e81cdd5d97a034135d1a3aff40665db46 WatchSource:0}: Error finding container 74c4eb12667d3bc8e90849892a290a5e81cdd5d97a034135d1a3aff40665db46: Status 404 returned error can't find the container with id 74c4eb12667d3bc8e90849892a290a5e81cdd5d97a034135d1a3aff40665db46
Sep 17 15:55:54.396665 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.396642    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9202fcd5_a26c_424e_a4c6_5031f21ec9bb.slice/crio-ac0b5a20d187280817bf6249336cd2547324fb5dffa993f54a83cf5526655022 WatchSource:0}: Error finding container ac0b5a20d187280817bf6249336cd2547324fb5dffa993f54a83cf5526655022: Status 404 returned error can't find the container with id ac0b5a20d187280817bf6249336cd2547324fb5dffa993f54a83cf5526655022
Sep 17 15:55:54.398042 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.398020    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01d3d75_6ebd_4acc_b209_50035e01489c.slice/crio-fb7ab429d3aaa942fb80a17a34f90350e270ece4319e4d7f8ccbdf2018113d6d WatchSource:0}: Error finding container fb7ab429d3aaa942fb80a17a34f90350e270ece4319e4d7f8ccbdf2018113d6d: Status 404 returned error can't find the container with id fb7ab429d3aaa942fb80a17a34f90350e270ece4319e4d7f8ccbdf2018113d6d
Sep 17 15:55:54.398836 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:55:54.398609    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43070168_32cf_48c0_8a1f_1211c0ee47dd.slice/crio-de82bb1555d854ede7eb18fc8c55ed6538cc60ae4f3462ec390254b001626cd1 WatchSource:0}: Error finding container de82bb1555d854ede7eb18fc8c55ed6538cc60ae4f3462ec390254b001626cd1: Status 404 returned error can't find the container with id de82bb1555d854ede7eb18fc8c55ed6538cc60ae4f3462ec390254b001626cd1
Sep 17 15:55:54.421759 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.421734    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:50:53 +0000 UTC" deadline="2028-03-19 04:22:26.427496283 +0000 UTC"
Sep 17 15:55:54.421759 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.421759    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13164h26m32.005740563s"
Sep 17 15:55:54.469439 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.469417    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:54.469549 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.469507    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:55:54.484120 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.484093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"ac0b5a20d187280817bf6249336cd2547324fb5dffa993f54a83cf5526655022"}
Sep 17 15:55:54.487553 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.487528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdfqz" event={"ID":"b2fc0529-7ca7-4545-829d-033b33052d9f","Type":"ContainerStarted","Data":"58fd66ccbc9bb7e78da18f53b30a2e9ff6afe94b9319be986415f4d074e3ba88"}
Sep 17 15:55:54.491888 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.491859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x9rwz" event={"ID":"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c","Type":"ContainerStarted","Data":"706ce49316fb185212faa30c83e4779872562cc7d6952f608180a07282865f8b"}
Sep 17 15:55:54.495453 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.495429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal" event={"ID":"0c3c972c2ec7ee80e8e1f20047dd8543","Type":"ContainerStarted","Data":"d23c0b27e47130a329bf3f17143241d46a1832d39131bf63b0eb2419d2054982"}
Sep 17 15:55:54.496775 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.496750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrkdd" event={"ID":"a87ffe04-0550-4937-9c43-f290c275f12f","Type":"ContainerStarted","Data":"74c4eb12667d3bc8e90849892a290a5e81cdd5d97a034135d1a3aff40665db46"}
Sep 17 15:55:54.500247 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.500225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lphmx" event={"ID":"095422d7-51d2-4c32-817f-f4ec7024face","Type":"ContainerStarted","Data":"0a7535ffd5a3974ac53573d7bb4bdc29598da4e512e58201b785515766db4ae1"}
Sep 17 15:55:54.503234 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.503209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvclt" event={"ID":"694737e1-5cd2-4739-8254-49c167544e79","Type":"ContainerStarted","Data":"15d778628038fda74e9062f1a5b4202cd168952ffeb16956931f89d7bcc2c989"}
Sep 17 15:55:54.506051 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.506031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797" event={"ID":"43070168-32cf-48c0-8a1f-1211c0ee47dd","Type":"ContainerStarted","Data":"de82bb1555d854ede7eb18fc8c55ed6538cc60ae4f3462ec390254b001626cd1"}
Sep 17 15:55:54.506996 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.506978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerStarted","Data":"fb7ab429d3aaa942fb80a17a34f90350e270ece4319e4d7f8ccbdf2018113d6d"}
Sep 17 15:55:54.511971 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.511930    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-187.ec2.internal" podStartSLOduration=2.511918655 podStartE2EDuration="2.511918655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:54.511568278 +0000 UTC m=+5.367268852" watchObservedRunningTime="2026-09-17 15:55:54.511918655 +0000 UTC m=+5.367619207"
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.970499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:54.970566    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.970785    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.970855    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:56.97083568 +0000 UTC m=+7.826536220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.971267    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:54.971407 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:54.971313    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:56.97129852 +0000 UTC m=+7.826999058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:55.071851 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.071811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:55.072031 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.071984    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:55.072031 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.072003    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:55.072031 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.072015    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:55.072195 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.072072    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:57.072053295 +0000 UTC m=+7.927753855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:55.422398 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.422284    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:50:53 +0000 UTC" deadline="2028-05-26 11:26:27.85279525 +0000 UTC"
Sep 17 15:55:55.422398 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.422326    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14803h30m32.430471971s"
Sep 17 15:55:55.471647 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.471081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:55.471647 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.471278    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:55:55.472166 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.471963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:55.472166 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:55.472088    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:55:55.520174 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.519945    2562 generic.go:356] "Generic (PLEG): container finished" podID="9264f1e21920152ed80f06b282288df9" containerID="b97c51f3e55a9b8a46155ca27600993cda3bb47ac77d1710fd3c8e13f6aa6dfc" exitCode=0
Sep 17 15:55:55.521283 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:55.521224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal" event={"ID":"9264f1e21920152ed80f06b282288df9","Type":"ContainerDied","Data":"b97c51f3e55a9b8a46155ca27600993cda3bb47ac77d1710fd3c8e13f6aa6dfc"}
Sep 17 15:55:56.469961 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:56.469936    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:56.470417 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:56.470055    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:55:56.532947 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:56.532903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal" event={"ID":"9264f1e21920152ed80f06b282288df9","Type":"ContainerStarted","Data":"d274878a5fc10d16cdd35e37fdb946174facebf1c75fd035b11b453e953a3cd8"}
Sep 17 15:55:56.988131 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:56.988096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:56.988313 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:56.988198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:56.988387 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:56.988374    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:56.988442 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:56.988431    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:00.988415016 +0000 UTC m=+11.844115567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:55:56.988844 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:56.988826    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:56.988932 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:56.988880    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:00.988863689 +0000 UTC m=+11.844564233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:55:57.089772 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:57.089441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:57.089772 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.089585    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:55:57.089772 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.089607    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:55:57.089772 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.089640    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:57.089772 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.089697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:56:01.089678524 +0000 UTC m=+11.945379079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:55:57.470833 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:57.470711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:57.471268 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.470856    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:55:57.471268 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:57.471200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:57.471363 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:57.471315    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:55:58.469475 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:58.469399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:55:58.469664 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:58.469551    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:55:59.469874 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:59.469821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:55:59.470316 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:59.469979    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:55:59.470810 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:55:59.470577    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:55:59.470810 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:55:59.470743    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:00.469305 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:00.469264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:00.469488 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:00.469390    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:01.022957 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:01.022913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:01.023405 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:01.022980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:01.023405 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.023091    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:01.023405 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.023118    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:01.023405 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.023164    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:09.023143081 +0000 UTC m=+19.878843626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:01.023405 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.023185    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:09.023175335 +0000 UTC m=+19.878875890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:01.123787 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:01.123748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:01.123963 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.123901    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:56:01.123963 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.123921    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:56:01.123963 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.123933    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:01.124131 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.123996    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:56:09.123977378 +0000 UTC m=+19.979677919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:01.470252 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:01.470128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:01.471706 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.471194    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:01.471706 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:01.471279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:01.471706 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:01.471368    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:02.470092 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:02.469996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:02.470605 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:02.470132    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:03.470115 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:03.470068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:03.470115 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:03.470115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:03.470569 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:03.470200    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:03.470569 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:03.470347    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:04.470149 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:04.470076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:04.470560 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:04.470191    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:05.469702 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:05.469663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:05.469925 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:05.469663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:05.469998 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:05.469816    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:05.469998 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:05.469947    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:06.469278 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:06.469238    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:06.469759 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:06.469396    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:07.470013 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:07.469983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:07.470433 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:07.470106    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:07.470433 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:07.470154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:07.470433 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:07.470261    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:08.469097 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:08.469061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:08.469340 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:08.469208    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:09.080768 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:09.080726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:09.081237 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:09.080800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:09.081237 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.080904    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:09.081237 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.080935    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:09.081237 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.080986    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:25.080964955 +0000 UTC m=+35.936665511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:09.081237 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.081009    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:25.080998231 +0000 UTC m=+35.936698770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:09.182445 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:09.182382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:09.182688 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.182669    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:56:09.182758 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.182696    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:56:09.182758 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.182710    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:09.182865 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.182779    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:56:25.182761522 +0000 UTC m=+36.038462062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:09.469372 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:09.469304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:09.469522 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.469429    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:09.469522 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:09.469484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:09.469608 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:09.469567    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:10.469808 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:10.469774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:10.470225 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:10.469892    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:11.469693 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:11.469673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:11.469799 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:11.469756    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:11.469859 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:11.469798    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:11.469859 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:11.469853    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:12.469319 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.469285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:12.469477 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:12.469410    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:12.525215 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.525191    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:56:12.563546 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.563519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lphmx" event={"ID":"095422d7-51d2-4c32-817f-f4ec7024face","Type":"ContainerStarted","Data":"baced4bf81e250d003aee19d25c385f9bd0f6d9114b9824760babeaa11b31402"}
Sep 17 15:56:12.564961 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.564931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvclt" event={"ID":"694737e1-5cd2-4739-8254-49c167544e79","Type":"ContainerStarted","Data":"dc0016687d9ce805cd44898103fe7d0b20c6a2e5c7101e1347a2754468827664"}
Sep 17 15:56:12.566602 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.566579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797" event={"ID":"43070168-32cf-48c0-8a1f-1211c0ee47dd","Type":"ContainerStarted","Data":"2510a19cd687520732f5aaa83c10b6f370857b2bdd2b87fdd443065f3f93d373"}
Sep 17 15:56:12.566709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.566650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797" event={"ID":"43070168-32cf-48c0-8a1f-1211c0ee47dd","Type":"ContainerStarted","Data":"a9896faf2596ff25c840120e5533b5beeff10bc9ef8b730bb27302bf25c08812"}
Sep 17 15:56:12.567974 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.567952    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="076b0e65e047e9770277948965cb76146e9fa0126a3b8ddd04577674684011dc" exitCode=0
Sep 17 15:56:12.568066 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.567982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"076b0e65e047e9770277948965cb76146e9fa0126a3b8ddd04577674684011dc"}
Sep 17 15:56:12.571096 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571081    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 15:56:12.571425 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571407    2562 generic.go:356] "Generic (PLEG): container finished" podID="9202fcd5-a26c-424e-a4c6-5031f21ec9bb" containerID="977a56b64311dd7d488c0228213ccdaedf068974a0535885fdcb403b386720f8" exitCode=1
Sep 17 15:56:12.571480 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"3b7ba1678fff02c7628058ac4f0ec84f654e9b1622e0edd5b0f5f516ebff1dec"}
Sep 17 15:56:12.571550 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"1414a31834d827baf8f3efbee4a9efe64e88aff42f636f7dbd65c43c19de29f0"}
Sep 17 15:56:12.571675 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"f50838cc5aa658e7ab574a496a2d242ba49140911ad888aaa662e176ff5dbebc"}
Sep 17 15:56:12.571675 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"4ac0c8b1b5cd6a03c07cd6dd9ab7318744c017476bbdaa39213539bb227b0ec8"}
Sep 17 15:56:12.571796 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerDied","Data":"977a56b64311dd7d488c0228213ccdaedf068974a0535885fdcb403b386720f8"}
Sep 17 15:56:12.571796 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.571745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"821adc46ec9d9afc86154c7dd554d099c21ae70ad6268e3f088e18310759f409"}
Sep 17 15:56:12.572896 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.572876    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdfqz" event={"ID":"b2fc0529-7ca7-4545-829d-033b33052d9f","Type":"ContainerStarted","Data":"77fd494ca939b0223535479b104f942b417125fc783b02ea395d337ecdc335cc"}
Sep 17 15:56:12.574250 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.574232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x9rwz" event={"ID":"15584b8e-f9c4-4f04-89b6-7ea3e4527a7c","Type":"ContainerStarted","Data":"d91d032713f4505a0c260b66400529774a4a7b604cb006fc2778a36d0543f797"}
Sep 17 15:56:12.583195 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.583157    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-187.ec2.internal" podStartSLOduration=20.583145694 podStartE2EDuration="20.583145694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:56.551754977 +0000 UTC m=+7.407455539" watchObservedRunningTime="2026-09-17 15:56:12.583145694 +0000 UTC m=+23.438846255"
Sep 17 15:56:12.583480 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.583461    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lphmx" podStartSLOduration=4.530500771 podStartE2EDuration="21.583456176s" totalImagesPullingTime="17.052955405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:12.583239765 +0000 UTC m=+23.438940317" watchObservedRunningTime="2026-09-17 15:56:12.583456176 +0000 UTC m=+23.439156736"
Sep 17 15:56:12.644848 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.644811    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zdfqz" podStartSLOduration=4.587026617 podStartE2EDuration="21.644799105s" totalImagesPullingTime="17.057772488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:12.611548864 +0000 UTC m=+23.467249425" watchObservedRunningTime="2026-09-17 15:56:12.644799105 +0000 UTC m=+23.500499645"
Sep 17 15:56:12.688951 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:12.688913    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x9rwz" podStartSLOduration=4.583025374 podStartE2EDuration="21.688899964s" totalImagesPullingTime="17.10587459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:12.668005639 +0000 UTC m=+23.523706200" watchObservedRunningTime="2026-09-17 15:56:12.688899964 +0000 UTC m=+23.544600503"
Sep 17 15:56:13.405716 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.405431    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:56:12.525208105Z","UUID":"111c395e-3e13-4327-b72d-666fa0aa2f1a","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:56:13.406973 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.406952    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:56:13.407075 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.406985    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:56:13.469712 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.469680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:13.469855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.469681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:13.469855 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:13.469817    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:13.469958 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:13.469900    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:13.578497 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.578405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrkdd" event={"ID":"a87ffe04-0550-4937-9c43-f290c275f12f","Type":"ContainerStarted","Data":"f20731ed8ad9d7994e7f407403b6e8b0dfa2b544df5f4a64bf8d54263b2ec1c7"}
Sep 17 15:56:13.581379 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.581346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797" event={"ID":"43070168-32cf-48c0-8a1f-1211c0ee47dd","Type":"ContainerStarted","Data":"43471c0da5dfb4066407cf9cd6f4ea1cc7dfbd8408edbf31c6bb0b0c24d4e7e1"}
Sep 17 15:56:13.596322 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.596276    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vvclt" podStartSLOduration=9.716646691 podStartE2EDuration="22.596258688s" totalImagesPullingTime="12.879611997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:12.689399411 +0000 UTC m=+23.545099971" watchObservedRunningTime="2026-09-17 15:56:13.596258688 +0000 UTC m=+24.451959250"
Sep 17 15:56:13.596529 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:13.596498    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrkdd" podStartSLOduration=5.546052813 podStartE2EDuration="22.596487842s" totalImagesPullingTime="17.050435029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:13.596406772 +0000 UTC m=+24.452107315" watchObservedRunningTime="2026-09-17 15:56:13.596487842 +0000 UTC m=+24.452188402"
Sep 17 15:56:14.469536 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:14.469502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:14.469733 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:14.469638    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:14.586690 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:14.586665    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 15:56:14.587103 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:14.587078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"09aec45a8d4b8b3298bbe26632a9abcfd03d529c4df7694fa6b27af0ce30ca0f"}
Sep 17 15:56:15.470149 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:15.470111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:15.470325 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:15.470111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:15.470325 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:15.470278    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:15.470475 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:15.470435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:15.703391 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:15.703360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:56:15.704142 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:15.704125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:56:15.725627 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:15.725540    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wj797" podStartSLOduration=6.14268532 podStartE2EDuration="24.725526323s" totalImagesPullingTime="18.582841003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:13.612654236 +0000 UTC m=+24.468354817" watchObservedRunningTime="2026-09-17 15:56:15.725526323 +0000 UTC m=+26.581226914"
Sep 17 15:56:16.469523 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.469495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:16.469712 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:16.469607    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:16.597197 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.597113    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 15:56:16.597743 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.597583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"6e8857300fb090d85cf96fdc3fc2b0413f55d072c40aa01e24933b16e11b4396"}
Sep 17 15:56:16.598060 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.598042    2562 scope.go:117] "RemoveContainer" containerID="977a56b64311dd7d488c0228213ccdaedf068974a0535885fdcb403b386720f8"
Sep 17 15:56:16.598428 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.598407    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:16.599025 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.598671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:56:16.599025 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.598701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:16.600212 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.599734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvclt"
Sep 17 15:56:16.616195 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.616039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:16.618934 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:16.618565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:17.469316 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.469133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:17.469775 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.469205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:17.469775 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:17.469385    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:17.469775 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:17.469503    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:17.600710 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.600680    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="8e5d908d61e4db4e5f34e9b545251caa736f45ce88c7e67341b9b28ecb01f106" exitCode=0
Sep 17 15:56:17.600834 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.600769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"8e5d908d61e4db4e5f34e9b545251caa736f45ce88c7e67341b9b28ecb01f106"}
Sep 17 15:56:17.604734 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.604718    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 15:56:17.605079 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.605057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" event={"ID":"9202fcd5-a26c-424e-a4c6-5031f21ec9bb","Type":"ContainerStarted","Data":"f36cdecd9b6150b72c42a1c36c2453b305a95dc9c6128da4aa980d50b6dcc22a"}
Sep 17 15:56:17.605159 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:17.605129    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:56:18.470069 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:18.470037    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:18.470495 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:18.470160    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:18.607689 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:18.607659    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:56:19.470140 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:19.470109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:19.470641 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:19.470242    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:19.470641 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:19.470295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:19.470641 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:19.470407    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:19.611329 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:19.611295    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="eeb3cc3182bf1791bc9333e037422195e542ae9cc3fdd99db5524b1d59528fd2" exitCode=0
Sep 17 15:56:19.611521 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:19.611352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"eeb3cc3182bf1791bc9333e037422195e542ae9cc3fdd99db5524b1d59528fd2"}
Sep 17 15:56:19.648186 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:19.648147    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" podStartSLOduration=11.576070666 podStartE2EDuration="28.648132493s" totalImagesPullingTime="17.072061827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:17.68932472 +0000 UTC m=+28.545025280" watchObservedRunningTime="2026-09-17 15:56:19.648132493 +0000 UTC m=+30.503833056"
Sep 17 15:56:20.469730 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:20.469697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:20.469883 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:20.469781    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:21.470470 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.470438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:21.471063 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:21.470531    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:21.471063 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.470637    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:21.471063 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:21.470753    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:21.617374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.617340    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="af48e61316f78610a6a069609eb4c9f9bc977e694b3b94477b86e1dffdfc924d" exitCode=0
Sep 17 15:56:21.617560 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.617414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"af48e61316f78610a6a069609eb4c9f9bc977e694b3b94477b86e1dffdfc924d"}
Sep 17 15:56:21.806708 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.806631    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:21.806868 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.806855    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:56:21.820450 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.820414    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" podUID="9202fcd5-a26c-424e-a4c6-5031f21ec9bb" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 15:56:21.828668 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:21.828637    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq" podUID="9202fcd5-a26c-424e-a4c6-5031f21ec9bb" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 15:56:22.469707 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:22.469674    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:22.469896 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:22.469792    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:23.469249 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:23.469214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:23.469907 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:23.469214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:23.469907 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:23.469364    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:23.469907 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:23.469416    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:24.469116 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:24.469071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:24.469295 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:24.469193    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:25.099020 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:25.098984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:25.099216 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:25.099040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:25.099216 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.099159    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:25.099336 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.099225    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:57.099205237 +0000 UTC m=+67.954905776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:56:25.099336 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.099167    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:25.099336 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.099306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret podName:249026f0-6f4d-43a2-96ae-bd00ac176d62 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:57.099290701 +0000 UTC m=+67.954991246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret") pod "global-pull-secret-syncer-xhmrn" (UID: "249026f0-6f4d-43a2-96ae-bd00ac176d62") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:56:25.199420 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:25.199382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:25.199606 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.199536    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:56:25.199606 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.199560    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:56:25.199606 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.199570    2562 projected.go:196] Error preparing data for projected volume kube-api-access-pmk74 for pod openshift-network-diagnostics/network-check-target-6pwd9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:25.199796 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.199650    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74 podName:49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b nodeName:}" failed. No retries permitted until 2026-09-17 15:56:57.199628935 +0000 UTC m=+68.055329489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmk74" (UniqueName: "kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74") pod "network-check-target-6pwd9" (UID: "49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:56:25.469589 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:25.469537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:25.470056 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.469680    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:25.470185 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:25.470069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:25.470248 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:25.470195    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:26.469621 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:26.469597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:26.469728 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:26.469700    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:26.632655 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:26.632609    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="07a9d1f7e890bda4450a8a61bab612eacc594afd713c86305c3f0345750cd648" exitCode=0
Sep 17 15:56:26.632791 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:26.632674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"07a9d1f7e890bda4450a8a61bab612eacc594afd713c86305c3f0345750cd648"}
Sep 17 15:56:27.469833 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:27.469805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:27.470207 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:27.469936    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:27.470207 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:27.469983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:27.470207 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:27.470061    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:27.637541 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:27.637511    2562 generic.go:356] "Generic (PLEG): container finished" podID="f01d3d75-6ebd-4acc-b209-50035e01489c" containerID="f8348b3ff7efaf4895c00fdcbe13fcd086433def819442933731d418383bb194" exitCode=0
Sep 17 15:56:27.637804 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:27.637569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerDied","Data":"f8348b3ff7efaf4895c00fdcbe13fcd086433def819442933731d418383bb194"}
Sep 17 15:56:28.469761 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:28.469711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:28.469967 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:28.469844    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:28.643547 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:28.643513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" event={"ID":"f01d3d75-6ebd-4acc-b209-50035e01489c","Type":"ContainerStarted","Data":"3eb8a97b77e0f73b76e09e1adab33a88f53227d7a1d34a967e6874d81e427499"}
Sep 17 15:56:28.666397 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:28.666350    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vbxj6" podStartSLOduration=9.964296136 podStartE2EDuration="37.666335553s" totalImagesPullingTime="27.702039417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:28.666310968 +0000 UTC m=+39.522011529" watchObservedRunningTime="2026-09-17 15:56:28.666335553 +0000 UTC m=+39.522036097"
Sep 17 15:56:29.469131 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:29.469090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:29.469304 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:29.469194    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:29.469304 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:29.469247    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:29.469443 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:29.469333    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:30.469289 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:30.469123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:30.469679 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:30.469354    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:31.470429 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:31.470394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:31.470784 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:31.470484    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:31.470784 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:31.470574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:31.470784 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:31.470672    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:32.469176 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:32.469149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:32.469338 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:32.469236    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:33.469909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:33.469877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:33.470370 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:33.469966    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:33.470370 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:33.470039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:33.470370 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:33.470121    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:34.469173 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:34.469142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:34.469452 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:34.469250    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:35.469105 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:35.469081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:35.469658 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:35.469183    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:35.469658 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:35.469274    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:35.469658 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:35.469399    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:36.469472 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:36.469440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:36.469874 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:36.469532    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:37.469513 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:37.469481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:37.469901 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:37.469589    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:37.469901 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:37.469646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:37.469901 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:37.469730    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:38.469320 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:38.469270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:38.469533 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:38.469401    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:39.469807 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:39.469778    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:39.469807 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:39.469789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:39.470241 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:39.469875    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:39.470241 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:39.469940    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:40.469697 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:40.469670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:40.469857 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:40.469760    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:41.469896 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:41.469864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:41.470326 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:41.469969    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:41.470326 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:41.470015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:41.470326 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:41.470092    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:42.469810 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:42.469778    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:42.470015 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:42.469886    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:43.469844 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:43.469812    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:43.470009 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:43.469935    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:43.470009 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:43.469980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:43.470287 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:43.470079    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:44.469548 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:44.469517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:44.469725 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:44.469627    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:45.470134 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:45.470103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:45.470490 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:45.470196    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:45.470490 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:45.470223    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:45.470490 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:45.470276    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:46.469843 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:46.469814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:46.470000 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:46.469901    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:47.469362 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:47.469331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:47.469362 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:47.469351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:47.469803 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:47.469424    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:47.469803 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:47.469571    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:48.469425 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:48.469393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:48.469806 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:48.469484    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:49.470006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.469970    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:49.470413 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:49.470078    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:49.470413 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.470110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:49.470413 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:49.470191    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:49.572742 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.572690    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b9nng"]
Sep 17 15:56:49.573550 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.573519    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhmrn"]
Sep 17 15:56:49.573699 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.573663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:49.573815 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:49.573788    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:49.574264 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.574240    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6pwd9"]
Sep 17 15:56:49.688781 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.688715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:49.688781 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:49.688744    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:49.688962 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:49.688821    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:49.688962 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:49.688938    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:51.469781 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:51.469755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:51.470314 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:51.469834    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:51.470314 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:51.469907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:51.470314 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:51.469992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:51.470314 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:51.470068    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:51.470314 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:51.469989    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:51.830936 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:51.830723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh6xq"
Sep 17 15:56:53.411594 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.411522    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-t8kxv"]
Sep 17 15:56:53.413245 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.413231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.416172 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.416152    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wzwjh\""
Sep 17 15:56:53.416777 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.416761    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:56:53.416832 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.416816    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:56:53.469712 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.469691    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:53.469712 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.469716    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:53.469849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.469694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:53.469849 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:53.469791    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmrn" podUID="249026f0-6f4d-43a2-96ae-bd00ac176d62"
Sep 17 15:56:53.469927 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:53.469856    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pwd9" podUID="49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b"
Sep 17 15:56:53.469927 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:53.469916    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b9nng" podUID="f948d78e-92e1-42d9-bc1a-be5e30b8c899"
Sep 17 15:56:53.495722 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.495699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlwk\" (UniqueName: \"kubernetes.io/projected/206bc29b-0e43-4446-9076-ad5705a7923b-kube-api-access-hhlwk\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.495811 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.495728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206bc29b-0e43-4446-9076-ad5705a7923b-tmp-dir\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.495811 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.495746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206bc29b-0e43-4446-9076-ad5705a7923b-hosts-file\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.596809 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.596788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlwk\" (UniqueName: \"kubernetes.io/projected/206bc29b-0e43-4446-9076-ad5705a7923b-kube-api-access-hhlwk\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.596931 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.596817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206bc29b-0e43-4446-9076-ad5705a7923b-tmp-dir\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.596931 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.596834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206bc29b-0e43-4446-9076-ad5705a7923b-hosts-file\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.597035 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.597001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/206bc29b-0e43-4446-9076-ad5705a7923b-hosts-file\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.597161 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.597142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206bc29b-0e43-4446-9076-ad5705a7923b-tmp-dir\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.616966 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.616950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlwk\" (UniqueName: \"kubernetes.io/projected/206bc29b-0e43-4446-9076-ad5705a7923b-kube-api-access-hhlwk\") pod \"node-resolver-t8kxv\" (UID: \"206bc29b-0e43-4446-9076-ad5705a7923b\") " pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.621563 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.621545    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-187.ec2.internal" event="NodeReady"
Sep 17 15:56:53.621687 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.621677    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:56:53.710151 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.710123    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j2qf2"]
Sep 17 15:56:53.712036 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.712018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.714578 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.714557    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bdqqj"]
Sep 17 15:56:53.715755 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.715732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:56:53.715876 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.715861    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrk56\""
Sep 17 15:56:53.716315 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.716296    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:56:53.716543 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.716528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.719216 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.719197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:56:53.719326 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.719308    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:56:53.721751 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.721265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxj9s\""
Sep 17 15:56:53.721751 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.721494    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:56:53.723220 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.723195    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8kxv"
Sep 17 15:56:53.732574 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:56:53.732553    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206bc29b_0e43_4446_9076_ad5705a7923b.slice/crio-aff41ea5e70918d8bee39bb38faeba75058fb61451520810893f83c650a3d69d WatchSource:0}: Error finding container aff41ea5e70918d8bee39bb38faeba75058fb61451520810893f83c650a3d69d: Status 404 returned error can't find the container with id aff41ea5e70918d8bee39bb38faeba75058fb61451520810893f83c650a3d69d
Sep 17 15:56:53.733350 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.733214    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2qf2"]
Sep 17 15:56:53.733932 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.733914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bdqqj"]
Sep 17 15:56:53.798127 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-tmp-dir\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.798248 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-config-volume\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.798248 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.798322 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c68\" (UniqueName: \"kubernetes.io/projected/557b9ce2-56a7-44c8-955c-ee140a6bcd00-kube-api-access-z9c68\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.798322 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557b9ce2-56a7-44c8-955c-ee140a6bcd00-cert\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.798322 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.798292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vg2s\" (UniqueName: \"kubernetes.io/projected/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-kube-api-access-7vg2s\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899180 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557b9ce2-56a7-44c8-955c-ee140a6bcd00-cert\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.899306 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vg2s\" (UniqueName: \"kubernetes.io/projected/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-kube-api-access-7vg2s\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899306 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-tmp-dir\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899306 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-config-volume\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899440 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899440 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c68\" (UniqueName: \"kubernetes.io/projected/557b9ce2-56a7-44c8-955c-ee140a6bcd00-kube-api-access-z9c68\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.899562 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:53.899434    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.899562 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:53.899518    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:56:54.399494699 +0000 UTC m=+65.255195250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:53.899720 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-tmp-dir\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.899864 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.899845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-config-volume\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:53.901687 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.901669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/557b9ce2-56a7-44c8-955c-ee140a6bcd00-cert\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.916645 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.916605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c68\" (UniqueName: \"kubernetes.io/projected/557b9ce2-56a7-44c8-955c-ee140a6bcd00-kube-api-access-z9c68\") pod \"ingress-canary-bdqqj\" (UID: \"557b9ce2-56a7-44c8-955c-ee140a6bcd00\") " pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:53.917296 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:53.917278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vg2s\" (UniqueName: \"kubernetes.io/projected/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-kube-api-access-7vg2s\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:54.029710 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.029639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bdqqj"
Sep 17 15:56:54.309277 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.309171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bdqqj"]
Sep 17 15:56:54.313805 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:56:54.313775    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557b9ce2_56a7_44c8_955c_ee140a6bcd00.slice/crio-694b1ba803987504d3caed2f713e9bce68704b168b980b09dc93f0b139821646 WatchSource:0}: Error finding container 694b1ba803987504d3caed2f713e9bce68704b168b980b09dc93f0b139821646: Status 404 returned error can't find the container with id 694b1ba803987504d3caed2f713e9bce68704b168b980b09dc93f0b139821646
Sep 17 15:56:54.402699 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.402667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:54.402889 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:54.402780    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:54.402889 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:54.402830    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:56:55.402817754 +0000 UTC m=+66.258518298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:54.700709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.700680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bdqqj" event={"ID":"557b9ce2-56a7-44c8-955c-ee140a6bcd00","Type":"ContainerStarted","Data":"694b1ba803987504d3caed2f713e9bce68704b168b980b09dc93f0b139821646"}
Sep 17 15:56:54.702126 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.702102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8kxv" event={"ID":"206bc29b-0e43-4446-9076-ad5705a7923b","Type":"ContainerStarted","Data":"1e5d4772f68e3a30fcf3be77241da0feb967f7cb1dbbdae83a6763ac05a45fa5"}
Sep 17 15:56:54.702222 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:54.702147    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8kxv" event={"ID":"206bc29b-0e43-4446-9076-ad5705a7923b","Type":"ContainerStarted","Data":"aff41ea5e70918d8bee39bb38faeba75058fb61451520810893f83c650a3d69d"}
Sep 17 15:56:55.408473 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.408434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:55.408681 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:55.408607    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:55.408745 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:55.408697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:56:57.408676115 +0000 UTC m=+68.264376656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:55.473685 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.473656    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:55.473849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.473775    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:55.473849 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.473809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:55.477191 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.477164    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:56:55.480274 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.480215    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:56:55.480482 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.480464    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:56:55.481242 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.481220    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fvxq9\""
Sep 17 15:56:55.481337 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.481260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v87nd\""
Sep 17 15:56:55.481497 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:55.481478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:56:57.121834 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.121790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:57.122292 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.121955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:56:57.122292 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:57.122077    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:56:57.122292 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:57.122149    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs podName:f948d78e-92e1-42d9-bc1a-be5e30b8c899 nodeName:}" failed. No retries permitted until 2026-09-17 15:58:01.12213099 +0000 UTC m=+131.977831537 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs") pod "network-metrics-daemon-b9nng" (UID: "f948d78e-92e1-42d9-bc1a-be5e30b8c899") : secret "metrics-daemon-secret" not found
Sep 17 15:56:57.125432 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.125413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/249026f0-6f4d-43a2-96ae-bd00ac176d62-original-pull-secret\") pod \"global-pull-secret-syncer-xhmrn\" (UID: \"249026f0-6f4d-43a2-96ae-bd00ac176d62\") " pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:57.223370 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.223184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:57.225727 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.225701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk74\" (UniqueName: \"kubernetes.io/projected/49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b-kube-api-access-pmk74\") pod \"network-check-target-6pwd9\" (UID: \"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b\") " pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:57.285789 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.285756    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmrn"
Sep 17 15:56:57.292718 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.292689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:56:57.424443 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.424408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:56:57.424592 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:57.424539    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:56:57.424651 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:56:57.424597    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:01.424581826 +0000 UTC m=+72.280282365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:56:57.554604 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.554508    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8kxv" podStartSLOduration=4.554489155 podStartE2EDuration="4.554489155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:54.721248004 +0000 UTC m=+65.576948584" watchObservedRunningTime="2026-09-17 15:56:57.554489155 +0000 UTC m=+68.410189715"
Sep 17 15:56:57.555111 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.554985    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhmrn"]
Sep 17 15:56:57.558737 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:56:57.558712    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249026f0_6f4d_43a2_96ae_bd00ac176d62.slice/crio-85d9a3650ec12e66504c6d9d74a13adfa04495c4391aa7f803e012d5c8fb1af6 WatchSource:0}: Error finding container 85d9a3650ec12e66504c6d9d74a13adfa04495c4391aa7f803e012d5c8fb1af6: Status 404 returned error can't find the container with id 85d9a3650ec12e66504c6d9d74a13adfa04495c4391aa7f803e012d5c8fb1af6
Sep 17 15:56:57.563344 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.563324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6pwd9"]
Sep 17 15:56:57.566876 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:56:57.566851    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a5a7d9_8bb0_4007_a2e8_c6f22a9bc75b.slice/crio-20f77b404749aa45cdd5e70c2d59e6c2ba5dbb7b66e3e3322381dc1d82345f44 WatchSource:0}: Error finding container 20f77b404749aa45cdd5e70c2d59e6c2ba5dbb7b66e3e3322381dc1d82345f44: Status 404 returned error can't find the container with id 20f77b404749aa45cdd5e70c2d59e6c2ba5dbb7b66e3e3322381dc1d82345f44
Sep 17 15:56:57.714039 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.714000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6pwd9" event={"ID":"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b","Type":"ContainerStarted","Data":"20f77b404749aa45cdd5e70c2d59e6c2ba5dbb7b66e3e3322381dc1d82345f44"}
Sep 17 15:56:57.715202 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.715177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhmrn" event={"ID":"249026f0-6f4d-43a2-96ae-bd00ac176d62","Type":"ContainerStarted","Data":"85d9a3650ec12e66504c6d9d74a13adfa04495c4391aa7f803e012d5c8fb1af6"}
Sep 17 15:56:57.716533 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.716517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bdqqj" event={"ID":"557b9ce2-56a7-44c8-955c-ee140a6bcd00","Type":"ContainerStarted","Data":"d956ce72487b388764008a82b97d021c562ffd558d2e182c7128c1f5e330dd4d"}
Sep 17 15:56:57.734437 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:56:57.734396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bdqqj" podStartSLOduration=2.162691433 podStartE2EDuration="4.734384754s" totalImagesPullingTime="2.571693321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:57.73433936 +0000 UTC m=+68.590039920" watchObservedRunningTime="2026-09-17 15:56:57.734384754 +0000 UTC m=+68.590085315"
Sep 17 15:57:01.453195 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:01.453162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:01.453882 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:01.453293    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:57:01.453882 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:01.453353    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:09.453334998 +0000 UTC m=+80.309035548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:57:02.731485 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:02.731341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6pwd9" event={"ID":"49a5a7d9-8bb0-4007-a2e8-c6f22a9bc75b","Type":"ContainerStarted","Data":"4dc6c7b4847872aba113e0c90122c149daa25c92b32e83dc50597045eac308d1"}
Sep 17 15:57:02.731485 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:02.731465    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:57:02.732868 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:02.732847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhmrn" event={"ID":"249026f0-6f4d-43a2-96ae-bd00ac176d62","Type":"ContainerStarted","Data":"97b3c544002580f8fda2e6c269e3431ae7ef4a8850b94c680322ebb2f744785a"}
Sep 17 15:57:02.747685 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:02.747640    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6pwd9" podStartSLOduration=67.203476989 podStartE2EDuration="1m11.74760554s" totalImagesPullingTime="4.544128551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:02.747248073 +0000 UTC m=+73.602948649" watchObservedRunningTime="2026-09-17 15:57:02.74760554 +0000 UTC m=+73.603306095"
Sep 17 15:57:07.658337 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.658278    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xhmrn" podStartSLOduration=72.103395361 podStartE2EDuration="1m16.658258181s" totalImagesPullingTime="4.55486282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:02.763805191 +0000 UTC m=+73.619505753" watchObservedRunningTime="2026-09-17 15:57:07.658258181 +0000 UTC m=+78.513958720"
Sep 17 15:57:07.659228 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.659207    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z2grg"]
Sep 17 15:57:07.661259 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.661245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.671896 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.671880    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:57:07.672164 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.672151    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:57:07.672834 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.672820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5b2qd\""
Sep 17 15:57:07.673244 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.673231    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:57:07.676534 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.676520    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:57:07.693468 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.693450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2grg"]
Sep 17 15:57:07.793180 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.793149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9204d550-e734-47cb-9a2b-f2b789f0b26a-crio-socket\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.793180 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.793184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9204d550-e734-47cb-9a2b-f2b789f0b26a-data-volume\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.793414 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.793224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmkg\" (UniqueName: \"kubernetes.io/projected/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-api-access-lcmkg\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.793414 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.793263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.793414 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.793330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893575 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893726 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9204d550-e734-47cb-9a2b-f2b789f0b26a-crio-socket\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893726 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9204d550-e734-47cb-9a2b-f2b789f0b26a-data-volume\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893726 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmkg\" (UniqueName: \"kubernetes.io/projected/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-api-access-lcmkg\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893726 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893726 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:07.893691    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:07.893944 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:07.893744    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.393729091 +0000 UTC m=+79.249429631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:07.893944 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:07.893772    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:07.893944 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9204d550-e734-47cb-9a2b-f2b789f0b26a-crio-socket\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.893944 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:07.893843    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:08.393827011 +0000 UTC m=+79.249527574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:07.894071 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.893964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9204d550-e734-47cb-9a2b-f2b789f0b26a-data-volume\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:07.917747 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:07.917698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmkg\" (UniqueName: \"kubernetes.io/projected/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-api-access-lcmkg\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:08.397851 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.397798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:08.398033 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.397875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:08.398033 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:08.397954    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:08.398033 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:08.397959    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:08.398033 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:08.398019    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:09.398004405 +0000 UTC m=+80.253704944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:08.398033 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:08.398032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:09.398026881 +0000 UTC m=+80.253727420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:08.423003 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.422979    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"]
Sep 17 15:57:08.425293 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.425279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"
Sep 17 15:57:08.427629 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.427585    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 15:57:08.427738 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.427687    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tjmch\""
Sep 17 15:57:08.428479 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.428464    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 15:57:08.437042 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.437019    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"]
Sep 17 15:57:08.498646 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.498610    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwsw\" (UniqueName: \"kubernetes.io/projected/e4e908d4-a734-42bb-aecd-1a746cdb7ee6-kube-api-access-qnwsw\") pod \"migrator-76794cbd6c-zts9m\" (UID: \"e4e908d4-a734-42bb-aecd-1a746cdb7ee6\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"
Sep 17 15:57:08.599319 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.599298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwsw\" (UniqueName: \"kubernetes.io/projected/e4e908d4-a734-42bb-aecd-1a746cdb7ee6-kube-api-access-qnwsw\") pod \"migrator-76794cbd6c-zts9m\" (UID: \"e4e908d4-a734-42bb-aecd-1a746cdb7ee6\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"
Sep 17 15:57:08.611135 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.611105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwsw\" (UniqueName: \"kubernetes.io/projected/e4e908d4-a734-42bb-aecd-1a746cdb7ee6-kube-api-access-qnwsw\") pod \"migrator-76794cbd6c-zts9m\" (UID: \"e4e908d4-a734-42bb-aecd-1a746cdb7ee6\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"
Sep 17 15:57:08.733918 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:08.733883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"
Sep 17 15:57:09.008448 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.008378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m"]
Sep 17 15:57:09.011652 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:09.011600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e908d4_a734_42bb_aecd_1a746cdb7ee6.slice/crio-093f99b817e0a9f53ce5acd98f972e48a994a19db3b86755047806c2d18542cf WatchSource:0}: Error finding container 093f99b817e0a9f53ce5acd98f972e48a994a19db3b86755047806c2d18542cf: Status 404 returned error can't find the container with id 093f99b817e0a9f53ce5acd98f972e48a994a19db3b86755047806c2d18542cf
Sep 17 15:57:09.019284 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.019263    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wnb2l"]
Sep 17 15:57:09.022143 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.022128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.025573 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.025553    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 15:57:09.025672 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.025640    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-txd4j\""
Sep 17 15:57:09.026448 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.026430    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 15:57:09.026520 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.026499    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 15:57:09.026569 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.026517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 15:57:09.034778 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.034759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wnb2l"]
Sep 17 15:57:09.102731 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.102706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6359c7e-e363-4331-afa8-6298f2936741-signing-cabundle\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.102855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.102759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6359c7e-e363-4331-afa8-6298f2936741-signing-key\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.102855 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.102787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcpx\" (UniqueName: \"kubernetes.io/projected/b6359c7e-e363-4331-afa8-6298f2936741-kube-api-access-llcpx\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.203535 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.203507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcpx\" (UniqueName: \"kubernetes.io/projected/b6359c7e-e363-4331-afa8-6298f2936741-kube-api-access-llcpx\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.203661 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.203570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6359c7e-e363-4331-afa8-6298f2936741-signing-cabundle\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.203709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.203662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6359c7e-e363-4331-afa8-6298f2936741-signing-key\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.204231 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.204203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6359c7e-e363-4331-afa8-6298f2936741-signing-cabundle\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.205985 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.205967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6359c7e-e363-4331-afa8-6298f2936741-signing-key\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.214486 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.214463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcpx\" (UniqueName: \"kubernetes.io/projected/b6359c7e-e363-4331-afa8-6298f2936741-kube-api-access-llcpx\") pod \"service-ca-55f6fc7b86-wnb2l\" (UID: \"b6359c7e-e363-4331-afa8-6298f2936741\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.330810 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.330737    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l"
Sep 17 15:57:09.405034 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.405008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:09.405209 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.405074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:09.405209 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.405165    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:09.405209 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.405197    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:09.405310 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.405223    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:11.4052076 +0000 UTC m=+82.260908160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:09.405310 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.405252    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:11.405239073 +0000 UTC m=+82.260939630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:09.506422 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.506397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:09.506585 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.506563    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:57:09.506664 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:09.506651    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls podName:d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:25.506628904 +0000 UTC m=+96.362329446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls") pod "dns-default-j2qf2" (UID: "d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a") : secret "dns-default-metrics-tls" not found
Sep 17 15:57:09.602738 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.602654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wnb2l"]
Sep 17 15:57:09.607397 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:09.607373    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6359c7e_e363_4331_afa8_6298f2936741.slice/crio-9c1aa725d5b23ed7fc95d1020e78bf692d9e86db7b255efc4a3ebdfe905f6e71 WatchSource:0}: Error finding container 9c1aa725d5b23ed7fc95d1020e78bf692d9e86db7b255efc4a3ebdfe905f6e71: Status 404 returned error can't find the container with id 9c1aa725d5b23ed7fc95d1020e78bf692d9e86db7b255efc4a3ebdfe905f6e71
Sep 17 15:57:09.751485 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.751447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m" event={"ID":"e4e908d4-a734-42bb-aecd-1a746cdb7ee6","Type":"ContainerStarted","Data":"093f99b817e0a9f53ce5acd98f972e48a994a19db3b86755047806c2d18542cf"}
Sep 17 15:57:09.752697 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.752673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l" event={"ID":"b6359c7e-e363-4331-afa8-6298f2936741","Type":"ContainerStarted","Data":"9c1aa725d5b23ed7fc95d1020e78bf692d9e86db7b255efc4a3ebdfe905f6e71"}
Sep 17 15:57:09.756514 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:09.756494    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8kxv_206bc29b-0e43-4446-9076-ad5705a7923b/dns-node-resolver/0.log"
Sep 17 15:57:10.356166 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:10.356084    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lphmx_095422d7-51d2-4c32-817f-f4ec7024face/node-ca/0.log"
Sep 17 15:57:10.758140 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:10.758108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m" event={"ID":"e4e908d4-a734-42bb-aecd-1a746cdb7ee6","Type":"ContainerStarted","Data":"6b4baf5621d44306e278b57503f580bd2e0f9d600e7e11af27b7309b5685bc21"}
Sep 17 15:57:10.758566 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:10.758164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m" event={"ID":"e4e908d4-a734-42bb-aecd-1a746cdb7ee6","Type":"ContainerStarted","Data":"78c272eda236744efbb6a1d7d47f1ca5b0ddadf1f930fbea96b4f5db20327904"}
Sep 17 15:57:10.777647 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:10.775088    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zts9m" podStartSLOduration=1.683922937 podStartE2EDuration="2.775059766s" totalImagesPullingTime="1.091136829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:10.773479613 +0000 UTC m=+81.629180194" watchObservedRunningTime="2026-09-17 15:57:10.775059766 +0000 UTC m=+81.630760330"
Sep 17 15:57:11.421675 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:11.421642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:11.421841 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:11.421732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:11.421841 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:11.421801    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:11.421930 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:11.421837    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:11.421930 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:11.421878    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:15.421856625 +0000 UTC m=+86.277557184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:11.421930 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:11.421897    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:15.421885483 +0000 UTC m=+86.277586022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:11.556987 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:11.556964    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bdqqj_557b9ce2-56a7-44c8-955c-ee140a6bcd00/serve-healthcheck-canary/0.log"
Sep 17 15:57:11.762890 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:11.762854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l" event={"ID":"b6359c7e-e363-4331-afa8-6298f2936741","Type":"ContainerStarted","Data":"0d4c8829f5d7adcd51cac1ce5c6a3cb4268ada254747f81141a1be47c739d826"}
Sep 17 15:57:11.777997 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:11.777949    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-wnb2l" podStartSLOduration=1.815794356 podStartE2EDuration="3.777934713s" totalImagesPullingTime="1.962140357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:11.777450368 +0000 UTC m=+82.633150948" watchObservedRunningTime="2026-09-17 15:57:11.777934713 +0000 UTC m=+82.633635274"
Sep 17 15:57:15.448663 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:15.448629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:15.449062 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:15.448684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:15.449062 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:15.448750    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:57:15.449062 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:15.448779    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:15.449062 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:15.448802    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:23.448788494 +0000 UTC m=+94.304489038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : configmap "kube-rbac-proxy" not found
Sep 17 15:57:15.449062 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:15.448849    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls podName:9204d550-e734-47cb-9a2b-f2b789f0b26a nodeName:}" failed. No retries permitted until 2026-09-17 15:57:23.448835546 +0000 UTC m=+94.304536084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2grg" (UID: "9204d550-e734-47cb-9a2b-f2b789f0b26a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:57:23.501131 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.501091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:23.501502 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.501153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:23.501778 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.501761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9204d550-e734-47cb-9a2b-f2b789f0b26a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:23.503427 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.503410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9204d550-e734-47cb-9a2b-f2b789f0b26a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2grg\" (UID: \"9204d550-e734-47cb-9a2b-f2b789f0b26a\") " pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:23.569568 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.569544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2grg"
Sep 17 15:57:23.865573 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:23.865547    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9204d550_e734_47cb_9a2b_f2b789f0b26a.slice/crio-9abf5c955950345c4b985291f4a9847bbbe005378f831ea7c8c6bcd0b9f60a40 WatchSource:0}: Error finding container 9abf5c955950345c4b985291f4a9847bbbe005378f831ea7c8c6bcd0b9f60a40: Status 404 returned error can't find the container with id 9abf5c955950345c4b985291f4a9847bbbe005378f831ea7c8c6bcd0b9f60a40
Sep 17 15:57:23.866457 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:23.866431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2grg"]
Sep 17 15:57:24.803845 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:24.803758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2grg" event={"ID":"9204d550-e734-47cb-9a2b-f2b789f0b26a","Type":"ContainerStarted","Data":"f111b01b46ffaf41fb99874839fd189524a5851b25ac296741cb0d52fb7cd79e"}
Sep 17 15:57:24.803845 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:24.803819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2grg" event={"ID":"9204d550-e734-47cb-9a2b-f2b789f0b26a","Type":"ContainerStarted","Data":"b51090ad1bc59b84a3f3d28660ea940bf78e71636d5aaad9f5ea8d78ba57c213"}
Sep 17 15:57:24.803845 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:24.803838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2grg" event={"ID":"9204d550-e734-47cb-9a2b-f2b789f0b26a","Type":"ContainerStarted","Data":"9abf5c955950345c4b985291f4a9847bbbe005378f831ea7c8c6bcd0b9f60a40"}
Sep 17 15:57:25.517375 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.517332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:25.520043 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.520005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a-metrics-tls\") pod \"dns-default-j2qf2\" (UID: \"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a\") " pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:25.522900 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.522876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:25.813999 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.813923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2grg" event={"ID":"9204d550-e734-47cb-9a2b-f2b789f0b26a","Type":"ContainerStarted","Data":"bf2101d0b00680bc65571a98c9b002c1ef42c35080b2817d0b7cd429c0417818"}
Sep 17 15:57:25.840779 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.840738    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z2grg" podStartSLOduration=17.149895457 podStartE2EDuration="18.840722355s" totalImagesPullingTime="1.690826898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:57:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:25.837481864 +0000 UTC m=+96.693182438" watchObservedRunningTime="2026-09-17 15:57:25.840722355 +0000 UTC m=+96.696422916"
Sep 17 15:57:25.926086 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:25.926062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2qf2"]
Sep 17 15:57:25.928227 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:25.928203    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3695e42_8632_4bd3_b3ac_7d2c6c1fac7a.slice/crio-4123d625246127552fa126801f15522112f3a5ddb9b1455077e6349107f656d2 WatchSource:0}: Error finding container 4123d625246127552fa126801f15522112f3a5ddb9b1455077e6349107f656d2: Status 404 returned error can't find the container with id 4123d625246127552fa126801f15522112f3a5ddb9b1455077e6349107f656d2
Sep 17 15:57:26.818775 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:26.818733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2qf2" event={"ID":"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a","Type":"ContainerStarted","Data":"4123d625246127552fa126801f15522112f3a5ddb9b1455077e6349107f656d2"}
Sep 17 15:57:27.824675 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:27.824642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2qf2" event={"ID":"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a","Type":"ContainerStarted","Data":"fbd0b8e673b2451305c46f7db58bd41a7b81d50b84cdb66a54885c9f5d3583ad"}
Sep 17 15:57:27.825214 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:27.824693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2qf2" event={"ID":"d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a","Type":"ContainerStarted","Data":"a3763dd800e5c92c5a5066fd3f3fe90c9d7da13cc946c2a706b62d06bda39839"}
Sep 17 15:57:27.825214 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:27.824800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:27.841947 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:27.841904    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j2qf2" podStartSLOduration=33.51665068 podStartE2EDuration="34.841890275s" totalImagesPullingTime="1.325239595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:27.841418395 +0000 UTC m=+98.697118974" watchObservedRunningTime="2026-09-17 15:57:27.841890275 +0000 UTC m=+98.697590836"
Sep 17 15:57:32.961808 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.961774    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675ff56f96-b2755"]
Sep 17 15:57:32.964644 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.964609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:32.969923 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.969902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:57:32.970429 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.970409    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:57:32.970534 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.970412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g5p99\""
Sep 17 15:57:32.970534 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.970446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:57:32.975739 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.975721    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:57:32.989186 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:32.989166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675ff56f96-b2755"]
Sep 17 15:57:33.068019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.067989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-certificates\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-bound-sa-token\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-image-registry-private-configuration\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-trusted-ca\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30a8a9a2-03df-4b84-9fdf-838bd98c5339-ca-trust-extracted\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068169 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4bx\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-kube-api-access-xw4bx\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068371 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-tls\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.068371 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.068245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-installation-pull-secrets\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.168949 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.168913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4bx\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-kube-api-access-xw4bx\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.168949 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.168956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-tls\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.168974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-installation-pull-secrets\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.168998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-certificates\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.169023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-bound-sa-token\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.169081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-image-registry-private-configuration\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.169112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-trusted-ca\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.169135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30a8a9a2-03df-4b84-9fdf-838bd98c5339-ca-trust-extracted\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.169654 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.169599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30a8a9a2-03df-4b84-9fdf-838bd98c5339-ca-trust-extracted\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.170035 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.170014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-certificates\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.170218 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.170196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30a8a9a2-03df-4b84-9fdf-838bd98c5339-trusted-ca\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.172123 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.172101    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-installation-pull-secrets\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.172123 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.172111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-registry-tls\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.172272 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.172172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30a8a9a2-03df-4b84-9fdf-838bd98c5339-image-registry-private-configuration\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.187539 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.187513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4bx\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-kube-api-access-xw4bx\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.211800 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.211771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30a8a9a2-03df-4b84-9fdf-838bd98c5339-bound-sa-token\") pod \"image-registry-675ff56f96-b2755\" (UID: \"30a8a9a2-03df-4b84-9fdf-838bd98c5339\") " pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.273763 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.273701    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.551069 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.550995    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675ff56f96-b2755"]
Sep 17 15:57:33.553758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:33.553728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a8a9a2_03df_4b84_9fdf_838bd98c5339.slice/crio-ebf1a5bdfbf3a68fb11bb8fe482b5d0eb189023dad9a68e2d5f2473524bafea2 WatchSource:0}: Error finding container ebf1a5bdfbf3a68fb11bb8fe482b5d0eb189023dad9a68e2d5f2473524bafea2: Status 404 returned error can't find the container with id ebf1a5bdfbf3a68fb11bb8fe482b5d0eb189023dad9a68e2d5f2473524bafea2
Sep 17 15:57:33.738170 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.738138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6pwd9"
Sep 17 15:57:33.848566 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.848493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff56f96-b2755" event={"ID":"30a8a9a2-03df-4b84-9fdf-838bd98c5339","Type":"ContainerStarted","Data":"cb3b21a69314c90257c2c8b4b194743345373f1fa747b55eeed6b1e05cba7f4f"}
Sep 17 15:57:33.848566 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.848536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff56f96-b2755" event={"ID":"30a8a9a2-03df-4b84-9fdf-838bd98c5339","Type":"ContainerStarted","Data":"ebf1a5bdfbf3a68fb11bb8fe482b5d0eb189023dad9a68e2d5f2473524bafea2"}
Sep 17 15:57:33.848760 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.848640    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:33.868390 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:33.868326    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-675ff56f96-b2755" podStartSLOduration=1.8683077240000001 podStartE2EDuration="1.868307724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:33.867920688 +0000 UTC m=+104.723621264" watchObservedRunningTime="2026-09-17 15:57:33.868307724 +0000 UTC m=+104.724008286"
Sep 17 15:57:37.830854 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:37.830826    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j2qf2"
Sep 17 15:57:38.478646 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.478598    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:57:38.481660 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.481644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.483953 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.483916    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:57:38.484101 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.483996    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:57:38.485032 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485010    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:57:38.485129 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lvx87\""
Sep 17 15:57:38.485190 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:57:38.485190 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485179    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:57:38.485257 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485222    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:57:38.485295 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.485224    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:57:38.489814 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.489791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:57:38.608851 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.608816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqrm\" (UniqueName: \"kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.609037 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.608922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.609037 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.608969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.609037 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.609007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.609037 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.609037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.609238 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.609059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710354 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710489 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqrm\" (UniqueName: \"kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710489 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710489 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710489 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.710717 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.710496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.711186 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.711164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.711301 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.711278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.711340 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.711278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.712848 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.712820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.712848 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.712830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.729012 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.728960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqrm\" (UniqueName: \"kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm\") pod \"console-fd8875844-zb74n\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") " pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:38.791231 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:38.791188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:39.058473 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:39.058400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:57:39.061758 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:39.061729    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a13a54_bb28_42b9_ae18_abe2073d7fc3.slice/crio-baef3f5968768384fb01cf87c361486d51eeb43c4a1836e945e3cb04f12efd76 WatchSource:0}: Error finding container baef3f5968768384fb01cf87c361486d51eeb43c4a1836e945e3cb04f12efd76: Status 404 returned error can't find the container with id baef3f5968768384fb01cf87c361486d51eeb43c4a1836e945e3cb04f12efd76
Sep 17 15:57:39.871751 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:39.871708    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8875844-zb74n" event={"ID":"05a13a54-bb28-42b9-ae18-abe2073d7fc3","Type":"ContainerStarted","Data":"baef3f5968768384fb01cf87c361486d51eeb43c4a1836e945e3cb04f12efd76"}
Sep 17 15:57:41.216020 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.215986    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2bvrr"]
Sep 17 15:57:41.218970 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.218942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.222152 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.221943    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:57:41.222152 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.221962    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t8vk7\""
Sep 17 15:57:41.222702 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.222680    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:57:41.222841 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.222822    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:57:41.222900 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.222886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:57:41.222987 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.222972    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:57:41.223039 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.222969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:57:41.329745 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-sys\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.329920 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.329920 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-textfile\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.329920 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.329920 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.329920 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-root\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.330168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9c9\" (UniqueName: \"kubernetes.io/projected/f717488d-bc7e-4e7f-b9e0-2391f83f1584-kube-api-access-tq9c9\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.330168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-wtmp\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.330168 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.329981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-metrics-client-ca\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431198 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-root\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9c9\" (UniqueName: \"kubernetes.io/projected/f717488d-bc7e-4e7f-b9e0-2391f83f1584-kube-api-access-tq9c9\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-wtmp\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-metrics-client-ca\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-root\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-sys\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431374 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-sys\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431698 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431698 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-textfile\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431698 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.431698 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.431905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-textfile\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:41.432003    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.432009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-metrics-client-ca\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.432035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-wtmp\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.432004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.432091 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:41.432062    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls podName:f717488d-bc7e-4e7f-b9e0-2391f83f1584 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:41.932043839 +0000 UTC m=+112.787744380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls") pod "node-exporter-2bvrr" (UID: "f717488d-bc7e-4e7f-b9e0-2391f83f1584") : secret "node-exporter-tls" not found
Sep 17 15:57:41.434237 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.434214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.441065 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.441025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9c9\" (UniqueName: \"kubernetes.io/projected/f717488d-bc7e-4e7f-b9e0-2391f83f1584-kube-api-access-tq9c9\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.934956 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.934921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:41.937293 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:41.937258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f717488d-bc7e-4e7f-b9e0-2391f83f1584-node-exporter-tls\") pod \"node-exporter-2bvrr\" (UID: \"f717488d-bc7e-4e7f-b9e0-2391f83f1584\") " pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:42.131221 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:42.131187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2bvrr"
Sep 17 15:57:42.967610 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:42.967566    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf717488d_bc7e_4e7f_b9e0_2391f83f1584.slice/crio-3be9ccf69cd04d9cd0e22f9c551e15adceec6ea74371b1f8f74c54e74f0b8f97 WatchSource:0}: Error finding container 3be9ccf69cd04d9cd0e22f9c551e15adceec6ea74371b1f8f74c54e74f0b8f97: Status 404 returned error can't find the container with id 3be9ccf69cd04d9cd0e22f9c551e15adceec6ea74371b1f8f74c54e74f0b8f97
Sep 17 15:57:43.257654 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.257555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"]
Sep 17 15:57:43.282320 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.282290    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"]
Sep 17 15:57:43.282485 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.282434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.285338 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.285310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:57:43.285763 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.285747    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fgr5w\""
Sep 17 15:57:43.285838 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.285797    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-25gepr711lgj1\""
Sep 17 15:57:43.286003 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.285990    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:57:43.286050 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.286016    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:57:43.286284 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.286267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:57:43.287550 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.287531    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:57:43.448498 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9gxx\" (UniqueName: \"kubernetes.io/projected/1d0aedbc-b968-461c-986d-2ca53c7900a8-kube-api-access-f9gxx\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d0aedbc-b968-461c-986d-2ca53c7900a8-metrics-client-ca\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-grpc-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.448764 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.449053 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.448837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550234 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9gxx\" (UniqueName: \"kubernetes.io/projected/1d0aedbc-b968-461c-986d-2ca53c7900a8-kube-api-access-f9gxx\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550234 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d0aedbc-b968-461c-986d-2ca53c7900a8-metrics-client-ca\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-grpc-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.550538 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.551141 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.550954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d0aedbc-b968-461c-986d-2ca53c7900a8-metrics-client-ca\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553443 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553533 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553533 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553731 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-grpc-tls\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553924 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.553962 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.553911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d0aedbc-b968-461c-986d-2ca53c7900a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.561271 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.561250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9gxx\" (UniqueName: \"kubernetes.io/projected/1d0aedbc-b968-461c-986d-2ca53c7900a8-kube-api-access-f9gxx\") pod \"thanos-querier-75dfbf546f-8kvg8\" (UID: \"1d0aedbc-b968-461c-986d-2ca53c7900a8\") " pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.591691 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.591670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:43.888673 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.888637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8875844-zb74n" event={"ID":"05a13a54-bb28-42b9-ae18-abe2073d7fc3","Type":"ContainerStarted","Data":"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"}
Sep 17 15:57:43.890227 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.890196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bvrr" event={"ID":"f717488d-bc7e-4e7f-b9e0-2391f83f1584","Type":"ContainerStarted","Data":"31ce88682905a4706a06ddf75b767dee5da017b536e4161fa58c8311d8c70d69"}
Sep 17 15:57:43.890335 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.890252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bvrr" event={"ID":"f717488d-bc7e-4e7f-b9e0-2391f83f1584","Type":"ContainerStarted","Data":"3be9ccf69cd04d9cd0e22f9c551e15adceec6ea74371b1f8f74c54e74f0b8f97"}
Sep 17 15:57:43.908691 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:43.908652    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd8875844-zb74n" podStartSLOduration=1.950910202 podStartE2EDuration="5.90863751s" totalImagesPullingTime="3.957727308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:43.908267929 +0000 UTC m=+114.763968522" watchObservedRunningTime="2026-09-17 15:57:43.90863751 +0000 UTC m=+114.764338062"
Sep 17 15:57:44.003122 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:44.003097    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"]
Sep 17 15:57:44.006400 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:44.006373    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0aedbc_b968_461c_986d_2ca53c7900a8.slice/crio-67e9d3d02e62acbaad9975f80a50bdb896f7dec4a4be860714b759405ab4bece WatchSource:0}: Error finding container 67e9d3d02e62acbaad9975f80a50bdb896f7dec4a4be860714b759405ab4bece: Status 404 returned error can't find the container with id 67e9d3d02e62acbaad9975f80a50bdb896f7dec4a4be860714b759405ab4bece
Sep 17 15:57:44.896017 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:44.895980    2562 generic.go:356] "Generic (PLEG): container finished" podID="f717488d-bc7e-4e7f-b9e0-2391f83f1584" containerID="31ce88682905a4706a06ddf75b767dee5da017b536e4161fa58c8311d8c70d69" exitCode=0
Sep 17 15:57:44.896201 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:44.896062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bvrr" event={"ID":"f717488d-bc7e-4e7f-b9e0-2391f83f1584","Type":"ContainerDied","Data":"31ce88682905a4706a06ddf75b767dee5da017b536e4161fa58c8311d8c70d69"}
Sep 17 15:57:44.897453 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:44.897421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"67e9d3d02e62acbaad9975f80a50bdb896f7dec4a4be860714b759405ab4bece"}
Sep 17 15:57:45.675019 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.674975    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-788bfb7cfb-d694t"]
Sep 17 15:57:45.677465 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.677440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.680405 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.680381    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gdt52\""
Sep 17 15:57:45.680965 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.680944    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:57:45.681087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.681024    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 15:57:45.681087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.681037    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7evi178ic0lf0\""
Sep 17 15:57:45.681196 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.681083    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 15:57:45.681307 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.681289    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 15:57:45.687811 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.687793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-788bfb7cfb-d694t"]
Sep 17 15:57:45.769749 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-client-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.769838 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.769838 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30257189-69fb-4251-a895-342acd26ffd7-audit-log\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.769945 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-client-certs\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.769945 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktznf\" (UniqueName: \"kubernetes.io/projected/30257189-69fb-4251-a895-342acd26ffd7-kube-api-access-ktznf\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.770040 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-tls\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.770040 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.769979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-metrics-server-audit-profiles\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870604 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-client-certs\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870736 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktznf\" (UniqueName: \"kubernetes.io/projected/30257189-69fb-4251-a895-342acd26ffd7-kube-api-access-ktznf\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870736 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-tls\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870736 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-metrics-server-audit-profiles\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870901 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-client-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870901 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.870901 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.870857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30257189-69fb-4251-a895-342acd26ffd7-audit-log\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.871440 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.871286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30257189-69fb-4251-a895-342acd26ffd7-audit-log\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.873010 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.872524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-metrics-server-audit-profiles\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.873010 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.872973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30257189-69fb-4251-a895-342acd26ffd7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.874218 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.874171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-client-certs\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.874389 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.874365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-secret-metrics-server-tls\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.875653 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.875634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30257189-69fb-4251-a895-342acd26ffd7-client-ca-bundle\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.888379 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.888337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktznf\" (UniqueName: \"kubernetes.io/projected/30257189-69fb-4251-a895-342acd26ffd7-kube-api-access-ktznf\") pod \"metrics-server-788bfb7cfb-d694t\" (UID: \"30257189-69fb-4251-a895-342acd26ffd7\") " pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:45.903993 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.903965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bvrr" event={"ID":"f717488d-bc7e-4e7f-b9e0-2391f83f1584","Type":"ContainerStarted","Data":"40ecbf1c55083bf84fbb22cbd2828722df74867981bcc03c77f316ca4afb5082"}
Sep 17 15:57:45.904096 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.904025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bvrr" event={"ID":"f717488d-bc7e-4e7f-b9e0-2391f83f1584","Type":"ContainerStarted","Data":"33c7e64aa016f9105713ec3096438f25c321c0c0c1fc9e8692b42f699aba35c9"}
Sep 17 15:57:45.906956 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.906925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"0c2a331566e63ae0e91aa55f9350beea07130a404d386425bfc7e45c3aae9499"}
Sep 17 15:57:45.907034 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.907001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"a66e2d37a02e9e175d78a391dd91321abcb42df0414381ca636f3c1b3e985403"}
Sep 17 15:57:45.907073 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.907043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"c0f4f16ae52d9f5861a7ec9a7e5030282adf79b09e8063ad7a34f185a7002e68"}
Sep 17 15:57:45.921770 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.921748    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"]
Sep 17 15:57:45.923880 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.923862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:45.926868 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.926807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:57:45.926954 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.926911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jjmcn\""
Sep 17 15:57:45.927392 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.927348    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2bvrr" podStartSLOduration=4.072914793 podStartE2EDuration="4.927335095s" totalImagesPullingTime="791.786959ms" totalInitContainerRuntime="62.633343ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:45.927286889 +0000 UTC m=+116.782987474" watchObservedRunningTime="2026-09-17 15:57:45.927335095 +0000 UTC m=+116.783035656"
Sep 17 15:57:45.934338 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.934319    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"]
Sep 17 15:57:45.987240 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:45.987217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:57:46.072316 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.072283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.072462 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.072342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgtf\" (UniqueName: \"kubernetes.io/projected/2d41b95c-913a-4160-bb29-4458d74a6681-kube-api-access-mhgtf\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.173230 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.173198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.173421 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.173242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgtf\" (UniqueName: \"kubernetes.io/projected/2d41b95c-913a-4160-bb29-4458d74a6681-kube-api-access-mhgtf\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.173421 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:46.173343    2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:57:46.173538 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:57:46.173432    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert podName:2d41b95c-913a-4160-bb29-4458d74a6681 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:46.673411396 +0000 UTC m=+117.529111942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-fmssx" (UID: "2d41b95c-913a-4160-bb29-4458d74a6681") : secret "monitoring-plugin-cert" not found
Sep 17 15:57:46.184432 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.184367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgtf\" (UniqueName: \"kubernetes.io/projected/2d41b95c-913a-4160-bb29-4458d74a6681-kube-api-access-mhgtf\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.255430 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.255406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-788bfb7cfb-d694t"]
Sep 17 15:57:46.259023 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:46.258995    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30257189_69fb_4251_a895_342acd26ffd7.slice/crio-d5338775f961b0d5a6dae8cc88d92441c9a7936c51148dd90e27ccebf5616fc4 WatchSource:0}: Error finding container d5338775f961b0d5a6dae8cc88d92441c9a7936c51148dd90e27ccebf5616fc4: Status 404 returned error can't find the container with id d5338775f961b0d5a6dae8cc88d92441c9a7936c51148dd90e27ccebf5616fc4
Sep 17 15:57:46.677457 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.677412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.679681 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.679659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d41b95c-913a-4160-bb29-4458d74a6681-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fmssx\" (UID: \"2d41b95c-913a-4160-bb29-4458d74a6681\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.813266 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.813204    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:57:46.820696 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.820675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.828995 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.828426    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:57:46.830572 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.830516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 15:57:46.833488 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.833310    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:46.915962 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.915911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"0ee3dcc057bdef5be8efab928a48d27dff69c283e8687873fe14d77b0a6946f5"}
Sep 17 15:57:46.916095 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.915973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"b25269df16bbee43eacf113859ec821741d4f8eaff4468bea087c0fb141c5f29"}
Sep 17 15:57:46.916095 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.916001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" event={"ID":"1d0aedbc-b968-461c-986d-2ca53c7900a8","Type":"ContainerStarted","Data":"0456bd570bead38397760f5b27c88ba91e5f3ff80c40c6d782778eb52551c96d"}
Sep 17 15:57:46.916214 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.916149    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:46.918645 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.918577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t" event={"ID":"30257189-69fb-4251-a895-342acd26ffd7","Type":"ContainerStarted","Data":"d5338775f961b0d5a6dae8cc88d92441c9a7936c51148dd90e27ccebf5616fc4"}
Sep 17 15:57:46.942753 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.942703    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8" podStartSLOduration=1.440021488 podStartE2EDuration="3.942685555s" totalImagesPullingTime="2.502664067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:46.941410392 +0000 UTC m=+117.797110954" watchObservedRunningTime="2026-09-17 15:57:46.942685555 +0000 UTC m=+117.798386118"
Sep 17 15:57:46.979795 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.979909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.979909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.979909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.980033 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979929    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.980033 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:46.980033 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:46.979997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqrll\" (UniqueName: \"kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081175 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081402 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081495 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081495 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081604 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqrll\" (UniqueName: \"kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081684 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.081739 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.081690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.082309 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.082100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.082486 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.082463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.082749 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.082717    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.082840 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.082749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.084749 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.084726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.084749 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.084740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.095031 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.095010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqrll\" (UniqueName: \"kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll\") pod \"console-7d9c58bd5b-tjfn2\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") " pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.119698 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.119674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"]
Sep 17 15:57:47.123377 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:47.123355    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d41b95c_913a_4160_bb29_4458d74a6681.slice/crio-faee09926aa2cebe8e0bbc435116261937975c283ba7e98c8d4237b3f6fd7059 WatchSource:0}: Error finding container faee09926aa2cebe8e0bbc435116261937975c283ba7e98c8d4237b3f6fd7059: Status 404 returned error can't find the container with id faee09926aa2cebe8e0bbc435116261937975c283ba7e98c8d4237b3f6fd7059
Sep 17 15:57:47.139829 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.139809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:47.431522 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.431486    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:57:47.627711 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:47.627647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d4b7c9_2d75_43d8_8e97_a67495c5d75c.slice/crio-814346ce70436326612cdd38049fe6ba67b07bb9dfc6f0093efdce92591551ee WatchSource:0}: Error finding container 814346ce70436326612cdd38049fe6ba67b07bb9dfc6f0093efdce92591551ee: Status 404 returned error can't find the container with id 814346ce70436326612cdd38049fe6ba67b07bb9dfc6f0093efdce92591551ee
Sep 17 15:57:47.924320 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.924226    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx" event={"ID":"2d41b95c-913a-4160-bb29-4458d74a6681","Type":"ContainerStarted","Data":"faee09926aa2cebe8e0bbc435116261937975c283ba7e98c8d4237b3f6fd7059"}
Sep 17 15:57:47.926163 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.926069    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t" event={"ID":"30257189-69fb-4251-a895-342acd26ffd7","Type":"ContainerStarted","Data":"4056980696ad431321de824b1f0e38b59dca507f1f13c17e2864a9831a5c299a"}
Sep 17 15:57:47.928034 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.928008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c58bd5b-tjfn2" event={"ID":"07d4b7c9-2d75-43d8-8e97-a67495c5d75c","Type":"ContainerStarted","Data":"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"}
Sep 17 15:57:47.928156 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.928048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c58bd5b-tjfn2" event={"ID":"07d4b7c9-2d75-43d8-8e97-a67495c5d75c","Type":"ContainerStarted","Data":"814346ce70436326612cdd38049fe6ba67b07bb9dfc6f0093efdce92591551ee"}
Sep 17 15:57:47.965709 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.965651    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d9c58bd5b-tjfn2" podStartSLOduration=1.965609792 podStartE2EDuration="1.965609792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:47.96477476 +0000 UTC m=+118.820475347" watchObservedRunningTime="2026-09-17 15:57:47.965609792 +0000 UTC m=+118.821310356"
Sep 17 15:57:47.966087 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:47.966054    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t" podStartSLOduration=1.5513776190000002 podStartE2EDuration="2.966045118s" totalImagesPullingTime="1.414667499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:47.946668672 +0000 UTC m=+118.802369233" watchObservedRunningTime="2026-09-17 15:57:47.966045118 +0000 UTC m=+118.821745678"
Sep 17 15:57:48.792284 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.792256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:48.792400 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.792305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:48.797238 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.797219    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:48.933556 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.933519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx" event={"ID":"2d41b95c-913a-4160-bb29-4458d74a6681","Type":"ContainerStarted","Data":"3fd361baf647a113b83bd6c330f33105133a32781e6dfda05b3a19fab11eebc5"}
Sep 17 15:57:48.934078 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.934055    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:48.937495 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.937475    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:57:48.938727 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.938707    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx"
Sep 17 15:57:48.952853 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:48.952783    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fmssx" podStartSLOduration=2.375894373 podStartE2EDuration="3.952766055s" totalImagesPullingTime="1.576871682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:48.951290625 +0000 UTC m=+119.806991210" watchObservedRunningTime="2026-09-17 15:57:48.952766055 +0000 UTC m=+119.808466618"
Sep 17 15:57:52.933553 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:52.933523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75dfbf546f-8kvg8"
Sep 17 15:57:54.855488 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:54.855457    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675ff56f96-b2755"
Sep 17 15:57:56.767071 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.767025    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:57:56.801961 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.801932    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:57:56.805955 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.805937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.816632 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.815081    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:57:56.864018 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.863990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864136 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864136 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864136 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864240 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864240 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.864240 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.864174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psc74\" (UniqueName: \"kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.964843 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.964843 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965048 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965048 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964907    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965048 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965048 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.964952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psc74\" (UniqueName: \"kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965048 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.965030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965757 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.965732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965882 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.965860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965926 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.965860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.965926 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.965900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.967327 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.967306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.967444 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.967425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:56.974376 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:56.974356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psc74\" (UniqueName: \"kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74\") pod \"console-7596f75778-q9lx9\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") " pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:57.120081 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.119999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:57:57.140780 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.140753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:57:57.392250 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.392217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:57:57.394725 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:57:57.394692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccb13c9_7607_4792_88a4_8df63feb364e.slice/crio-feb4244bb55714717eee49aac2ab962932ec0f76a25bbd0df80ad3e7b042f9a0 WatchSource:0}: Error finding container feb4244bb55714717eee49aac2ab962932ec0f76a25bbd0df80ad3e7b042f9a0: Status 404 returned error can't find the container with id feb4244bb55714717eee49aac2ab962932ec0f76a25bbd0df80ad3e7b042f9a0
Sep 17 15:57:57.965924 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.965887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7596f75778-q9lx9" event={"ID":"cccb13c9-7607-4792-88a4-8df63feb364e","Type":"ContainerStarted","Data":"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"}
Sep 17 15:57:57.966405 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.965940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7596f75778-q9lx9" event={"ID":"cccb13c9-7607-4792-88a4-8df63feb364e","Type":"ContainerStarted","Data":"feb4244bb55714717eee49aac2ab962932ec0f76a25bbd0df80ad3e7b042f9a0"}
Sep 17 15:57:57.990938 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:57:57.990892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7596f75778-q9lx9" podStartSLOduration=1.9908781960000002 podStartE2EDuration="1.990878196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:57.988031078 +0000 UTC m=+128.843731652" watchObservedRunningTime="2026-09-17 15:57:57.990878196 +0000 UTC m=+128.846578758"
Sep 17 15:58:01.195066 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.195036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:58:01.197381 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.197356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f948d78e-92e1-42d9-bc1a-be5e30b8c899-metrics-certs\") pod \"network-metrics-daemon-b9nng\" (UID: \"f948d78e-92e1-42d9-bc1a-be5e30b8c899\") " pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:58:01.201384 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.201368    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v87nd\""
Sep 17 15:58:01.209267 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.209249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b9nng"
Sep 17 15:58:01.484314 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.484237    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b9nng"]
Sep 17 15:58:01.487235 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:58:01.487209    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf948d78e_92e1_42d9_bc1a_be5e30b8c899.slice/crio-6bdb609f35bbb9de5613a179e9ff7f21f8cc78627b11b19f09487d2085154654 WatchSource:0}: Error finding container 6bdb609f35bbb9de5613a179e9ff7f21f8cc78627b11b19f09487d2085154654: Status 404 returned error can't find the container with id 6bdb609f35bbb9de5613a179e9ff7f21f8cc78627b11b19f09487d2085154654
Sep 17 15:58:01.982360 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:01.982317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9nng" event={"ID":"f948d78e-92e1-42d9-bc1a-be5e30b8c899","Type":"ContainerStarted","Data":"6bdb609f35bbb9de5613a179e9ff7f21f8cc78627b11b19f09487d2085154654"}
Sep 17 15:58:02.988113 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:02.988076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9nng" event={"ID":"f948d78e-92e1-42d9-bc1a-be5e30b8c899","Type":"ContainerStarted","Data":"2cddf08fcda2df9243c638e9f8af352301bdd32e66b0360473d9940534ac9d18"}
Sep 17 15:58:02.988463 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:02.988134    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b9nng" event={"ID":"f948d78e-92e1-42d9-bc1a-be5e30b8c899","Type":"ContainerStarted","Data":"399aaae7fcca7695d43b2f52d8f817f9c9ef8a69da874ef8fbfba9b89832c38e"}
Sep 17 15:58:03.004283 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:03.004235    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b9nng" podStartSLOduration=131.187638648 podStartE2EDuration="2m12.004220003s" totalImagesPullingTime="816.581355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:03.002938511 +0000 UTC m=+133.858639097" watchObservedRunningTime="2026-09-17 15:58:03.004220003 +0000 UTC m=+133.859920600"
Sep 17 15:58:05.987921 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:05.987882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:58:05.988403 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:05.987956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:58:07.121124 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:07.121089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:58:07.121556 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:07.121151    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:58:07.125964 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:07.125941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:58:08.009472 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:08.009446    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:58:08.057018 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:08.056991    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:58:21.785430 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:21.785394    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d9c58bd5b-tjfn2" podUID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" containerName="console" containerID="cri-o://0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a" gracePeriod=15
Sep 17 15:58:22.029909 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.029880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:58:22.060754 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.060692    2562 generic.go:356] "Generic (PLEG): container finished" podID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" containerID="0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a" exitCode=0
Sep 17 15:58:22.060754 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.060740    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c58bd5b-tjfn2"
Sep 17 15:58:22.060908 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.060767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c58bd5b-tjfn2" event={"ID":"07d4b7c9-2d75-43d8-8e97-a67495c5d75c","Type":"ContainerDied","Data":"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"}
Sep 17 15:58:22.060908 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.060828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c58bd5b-tjfn2" event={"ID":"07d4b7c9-2d75-43d8-8e97-a67495c5d75c","Type":"ContainerDied","Data":"814346ce70436326612cdd38049fe6ba67b07bb9dfc6f0093efdce92591551ee"}
Sep 17 15:58:22.060908 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.060847    2562 scope.go:117] "RemoveContainer" containerID="0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"
Sep 17 15:58:22.070104 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.070078    2562 scope.go:117] "RemoveContainer" containerID="0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"
Sep 17 15:58:22.070370 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:58:22.070349    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a\": container with ID starting with 0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a not found: ID does not exist" containerID="0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"
Sep 17 15:58:22.070455 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.070378    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a"} err="failed to get container status \"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a\": rpc error: code = NotFound desc = could not find container \"0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a\": container with ID starting with 0d55343ee31d583f710b4fd6784de8a447f6396af19c7415c100e678b9f88c1a not found: ID does not exist"
Sep 17 15:58:22.150886 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.150845    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqrll\" (UniqueName: \"kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.150886 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.150889    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151102 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.150914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151102 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.150948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151102 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151073    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151256 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151110    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151256 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151137    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert\") pod \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\" (UID: \"07d4b7c9-2d75-43d8-8e97-a67495c5d75c\") "
Sep 17 15:58:22.151421 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:22.151502 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151478    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-trusted-ca-bundle\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.151565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config" (OuterVolumeSpecName: "console-config") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:22.151634 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151558    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:22.151804 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.151782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca" (OuterVolumeSpecName: "service-ca") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:22.153274 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.153245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll" (OuterVolumeSpecName: "kube-api-access-sqrll") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "kube-api-access-sqrll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:22.153274 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.153259    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:22.153391 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.153292    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "07d4b7c9-2d75-43d8-8e97-a67495c5d75c" (UID: "07d4b7c9-2d75-43d8-8e97-a67495c5d75c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:22.252601 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252553    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqrll\" (UniqueName: \"kubernetes.io/projected/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-kube-api-access-sqrll\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.252601 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252593    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.252601 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252604    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-service-ca\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.252601 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252633    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-oauth-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.252601 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252643    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-console-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.252910 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.252651    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07d4b7c9-2d75-43d8-8e97-a67495c5d75c-oauth-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:22.387678 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.387607    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:58:22.390931 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:22.390912    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d9c58bd5b-tjfn2"]
Sep 17 15:58:23.473779 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:23.473746    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" path="/var/lib/kubelet/pods/07d4b7c9-2d75-43d8-8e97-a67495c5d75c/volumes"
Sep 17 15:58:25.992443 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:25.992413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:58:25.996212 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:25.996192    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-788bfb7cfb-d694t"
Sep 17 15:58:33.076232 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.076143    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fd8875844-zb74n" podUID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" containerName="console" containerID="cri-o://2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76" gracePeriod=15
Sep 17 15:58:33.324783 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.324755    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:58:33.439411 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439389    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jqrm\" (UniqueName: \"kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439480    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439506    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439565 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert\") pod \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\" (UID: \"05a13a54-bb28-42b9-ae18-abe2073d7fc3\") "
Sep 17 15:58:33.439943 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca" (OuterVolumeSpecName: "service-ca") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:33.439943 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439892    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config" (OuterVolumeSpecName: "console-config") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:33.439943 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.439920    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:33.441756 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.441726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:33.441848 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.441768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm" (OuterVolumeSpecName: "kube-api-access-9jqrm") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "kube-api-access-9jqrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:33.442290 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.442267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05a13a54-bb28-42b9-ae18-abe2073d7fc3" (UID: "05a13a54-bb28-42b9-ae18-abe2073d7fc3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:33.540551 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540523    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-service-ca\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.540551 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540549    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-oauth-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.540551 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540558    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jqrm\" (UniqueName: \"kubernetes.io/projected/05a13a54-bb28-42b9-ae18-abe2073d7fc3-kube-api-access-9jqrm\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.540777 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540567    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.540777 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540577    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a13a54-bb28-42b9-ae18-abe2073d7fc3-oauth-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.540777 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:33.540585    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a13a54-bb28-42b9-ae18-abe2073d7fc3-console-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:58:34.109852 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.109815    2562 generic.go:356] "Generic (PLEG): container finished" podID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" containerID="2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76" exitCode=0
Sep 17 15:58:34.110282 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.109890    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8875844-zb74n"
Sep 17 15:58:34.110282 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.109929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8875844-zb74n" event={"ID":"05a13a54-bb28-42b9-ae18-abe2073d7fc3","Type":"ContainerDied","Data":"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"}
Sep 17 15:58:34.110282 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.109983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8875844-zb74n" event={"ID":"05a13a54-bb28-42b9-ae18-abe2073d7fc3","Type":"ContainerDied","Data":"baef3f5968768384fb01cf87c361486d51eeb43c4a1836e945e3cb04f12efd76"}
Sep 17 15:58:34.110282 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.110014    2562 scope.go:117] "RemoveContainer" containerID="2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"
Sep 17 15:58:34.119128 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.119108    2562 scope.go:117] "RemoveContainer" containerID="2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"
Sep 17 15:58:34.119367 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:58:34.119348    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76\": container with ID starting with 2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76 not found: ID does not exist" containerID="2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"
Sep 17 15:58:34.119420 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.119374    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76"} err="failed to get container status \"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76\": rpc error: code = NotFound desc = could not find container \"2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76\": container with ID starting with 2f573f28ded106ed1e21e382a4c90f3233b8eef618191a07f0d5378835467c76 not found: ID does not exist"
Sep 17 15:58:34.159467 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.159444    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:58:34.168503 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:34.168466    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd8875844-zb74n"]
Sep 17 15:58:35.474314 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:58:35.474281    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" path="/var/lib/kubelet/pods/05a13a54-bb28-42b9-ae18-abe2073d7fc3/volumes"
Sep 17 15:59:16.187629 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187579    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187845    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" containerName="console"
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" containerName="console"
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187870    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" containerName="console"
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187875    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" containerName="console"
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187921    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a13a54-bb28-42b9-ae18-abe2073d7fc3" containerName="console"
Sep 17 15:59:16.188074 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.187929    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d4b7c9-2d75-43d8-8e97-a67495c5d75c" containerName="console"
Sep 17 15:59:16.189818 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.189802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.203138 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.203113    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 15:59:16.267595 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267595 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvr9\" (UniqueName: \"kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.267813 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.267812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.368992 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.368957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.368992 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.368996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369243 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvr9\" (UniqueName: \"kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369996 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.369996 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.370129 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.369992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.370129 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.370026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.372125 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.372093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.372213 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.372167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.379889 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.379866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvr9\" (UniqueName: \"kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9\") pod \"console-5748cc9774-p8wg7\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") " pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.498526 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.498495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:16.775055 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:16.774977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 15:59:16.778319 ip-10-0-143-187 kubenswrapper[2562]: W0917 15:59:16.778293    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b07fa26_a82b_48fb_8ea4_321d7aeef7bd.slice/crio-956e85c3ef1e0323f0ac83fe6b5141b37c3c9ba3e6289cdb382882a7c830e840 WatchSource:0}: Error finding container 956e85c3ef1e0323f0ac83fe6b5141b37c3c9ba3e6289cdb382882a7c830e840: Status 404 returned error can't find the container with id 956e85c3ef1e0323f0ac83fe6b5141b37c3c9ba3e6289cdb382882a7c830e840
Sep 17 15:59:17.278837 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:17.278803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748cc9774-p8wg7" event={"ID":"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd","Type":"ContainerStarted","Data":"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"}
Sep 17 15:59:17.279218 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:17.278851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748cc9774-p8wg7" event={"ID":"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd","Type":"ContainerStarted","Data":"956e85c3ef1e0323f0ac83fe6b5141b37c3c9ba3e6289cdb382882a7c830e840"}
Sep 17 15:59:17.304133 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:17.304082    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5748cc9774-p8wg7" podStartSLOduration=1.3040678620000001 podStartE2EDuration="1.304067862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:17.303451777 +0000 UTC m=+208.159152379" watchObservedRunningTime="2026-09-17 15:59:17.304067862 +0000 UTC m=+208.159768422"
Sep 17 15:59:26.498683 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:26.498645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:26.499117 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:26.498703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:26.503404 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:26.503380    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:27.318096 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:27.318068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 15:59:27.391551 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:27.391519    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:59:52.410390 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.410347    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7596f75778-q9lx9" podUID="cccb13c9-7607-4792-88a4-8df63feb364e" containerName="console" containerID="cri-o://a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54" gracePeriod=15
Sep 17 15:59:52.652773 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.652747    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:59:52.742832 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742807    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742919    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psc74\" (UniqueName: \"kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743006 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.742976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743265 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.743010    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config\") pod \"cccb13c9-7607-4792-88a4-8df63feb364e\" (UID: \"cccb13c9-7607-4792-88a4-8df63feb364e\") "
Sep 17 15:59:52.743331 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.743199    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:52.743418 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.743388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca" (OuterVolumeSpecName: "service-ca") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:52.743524 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.743428    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config" (OuterVolumeSpecName: "console-config") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:52.743524 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.743441    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:52.744967 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.744947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:52.745064 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.745037    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:52.745194 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.745165    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74" (OuterVolumeSpecName: "kube-api-access-psc74") pod "cccb13c9-7607-4792-88a4-8df63feb364e" (UID: "cccb13c9-7607-4792-88a4-8df63feb364e"). InnerVolumeSpecName "kube-api-access-psc74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:52.843741 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843708    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-trusted-ca-bundle\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.843741 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843732    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-service-ca\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.843741 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843741    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psc74\" (UniqueName: \"kubernetes.io/projected/cccb13c9-7607-4792-88a4-8df63feb364e-kube-api-access-psc74\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.843741 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843750    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.844000 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843758    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-console-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.844000 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843766    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cccb13c9-7607-4792-88a4-8df63feb364e-oauth-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:52.844000 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:52.843774    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cccb13c9-7607-4792-88a4-8df63feb364e-console-oauth-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 15:59:53.408526 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.408494    2562 generic.go:356] "Generic (PLEG): container finished" podID="cccb13c9-7607-4792-88a4-8df63feb364e" containerID="a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54" exitCode=0
Sep 17 15:59:53.408707 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.408540    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7596f75778-q9lx9"
Sep 17 15:59:53.408707 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.408571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7596f75778-q9lx9" event={"ID":"cccb13c9-7607-4792-88a4-8df63feb364e","Type":"ContainerDied","Data":"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"}
Sep 17 15:59:53.408707 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.408611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7596f75778-q9lx9" event={"ID":"cccb13c9-7607-4792-88a4-8df63feb364e","Type":"ContainerDied","Data":"feb4244bb55714717eee49aac2ab962932ec0f76a25bbd0df80ad3e7b042f9a0"}
Sep 17 15:59:53.408707 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.408644    2562 scope.go:117] "RemoveContainer" containerID="a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"
Sep 17 15:59:53.417110 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.416939    2562 scope.go:117] "RemoveContainer" containerID="a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"
Sep 17 15:59:53.417318 ip-10-0-143-187 kubenswrapper[2562]: E0917 15:59:53.417196    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54\": container with ID starting with a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54 not found: ID does not exist" containerID="a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"
Sep 17 15:59:53.417318 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.417216    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54"} err="failed to get container status \"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54\": rpc error: code = NotFound desc = could not find container \"a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54\": container with ID starting with a3fe14309c9933c83603514f52e3fdd8ecb5021cb3c646de83e22e085901cf54 not found: ID does not exist"
Sep 17 15:59:53.451991 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.451967    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:59:53.463092 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.463071    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7596f75778-q9lx9"]
Sep 17 15:59:53.474105 ip-10-0-143-187 kubenswrapper[2562]: I0917 15:59:53.474086    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccb13c9-7607-4792-88a4-8df63feb364e" path="/var/lib/kubelet/pods/cccb13c9-7607-4792-88a4-8df63feb364e/volumes"
Sep 17 16:00:44.215141 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.215104    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"]
Sep 17 16:00:44.215591 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.215386    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccb13c9-7607-4792-88a4-8df63feb364e" containerName="console"
Sep 17 16:00:44.215591 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.215397    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccb13c9-7607-4792-88a4-8df63feb364e" containerName="console"
Sep 17 16:00:44.215591 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.215454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccb13c9-7607-4792-88a4-8df63feb364e" containerName="console"
Sep 17 16:00:44.217455 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.217439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.222283 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.222262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:00:44.222366 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.222262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:00:44.223976 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.223658    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-22ddr\""
Sep 17 16:00:44.236216 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.236195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"]
Sep 17 16:00:44.308056 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.308026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.308187 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.308070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf996\" (UniqueName: \"kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.308187 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.308103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.408991 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.408961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.409107 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.409013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf996\" (UniqueName: \"kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.409107 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.409038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.409395 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.409377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.409455 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.409434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.428503 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.428468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf996\" (UniqueName: \"kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.526154 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.526073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:00:44.805086 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:44.805014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"]
Sep 17 16:00:44.809004 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:00:44.808978    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595fe665_fe7e_41f6_860d_280aa99be2fa.slice/crio-7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b WatchSource:0}: Error finding container 7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b: Status 404 returned error can't find the container with id 7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b
Sep 17 16:00:45.594184 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:45.594152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4" event={"ID":"595fe665-fe7e-41f6-860d-280aa99be2fa","Type":"ContainerStarted","Data":"7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b"}
Sep 17 16:00:52.386534 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.386511    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:00:52.386976 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.386683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:00:52.390388 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.390370    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:00:52.621073 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.620980    2562 generic.go:356] "Generic (PLEG): container finished" podID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerID="3ac0269380598f8e644d6a81d06249c16eb71d76859b7730dd22233d923371e8" exitCode=0
Sep 17 16:00:52.621073 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.621064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4" event={"ID":"595fe665-fe7e-41f6-860d-280aa99be2fa","Type":"ContainerDied","Data":"3ac0269380598f8e644d6a81d06249c16eb71d76859b7730dd22233d923371e8"}
Sep 17 16:00:52.622057 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:52.622043    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:00:56.636740 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:56.636707    2562 generic.go:356] "Generic (PLEG): container finished" podID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerID="e279e0be5376e300b30df3b01a07de8da63419c3cd78f21f0cd007c6729df4f7" exitCode=0
Sep 17 16:00:56.637188 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:00:56.636776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4" event={"ID":"595fe665-fe7e-41f6-860d-280aa99be2fa","Type":"ContainerDied","Data":"e279e0be5376e300b30df3b01a07de8da63419c3cd78f21f0cd007c6729df4f7"}
Sep 17 16:01:03.666456 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:03.666371    2562 generic.go:356] "Generic (PLEG): container finished" podID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerID="d68f5749819e3d865f73eb7c9e4d5d3ab96be1a219e02e7c0a35edda02e9f1a1" exitCode=0
Sep 17 16:01:03.666456 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:03.666429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4" event={"ID":"595fe665-fe7e-41f6-860d-280aa99be2fa","Type":"ContainerDied","Data":"d68f5749819e3d865f73eb7c9e4d5d3ab96be1a219e02e7c0a35edda02e9f1a1"}
Sep 17 16:01:04.794873 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.794848    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:01:04.877499 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.877470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle\") pod \"595fe665-fe7e-41f6-860d-280aa99be2fa\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") "
Sep 17 16:01:04.877682 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.877514    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf996\" (UniqueName: \"kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996\") pod \"595fe665-fe7e-41f6-860d-280aa99be2fa\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") "
Sep 17 16:01:04.877682 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.877552    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util\") pod \"595fe665-fe7e-41f6-860d-280aa99be2fa\" (UID: \"595fe665-fe7e-41f6-860d-280aa99be2fa\") "
Sep 17 16:01:04.878143 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.878116    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle" (OuterVolumeSpecName: "bundle") pod "595fe665-fe7e-41f6-860d-280aa99be2fa" (UID: "595fe665-fe7e-41f6-860d-280aa99be2fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:04.879696 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.879667    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996" (OuterVolumeSpecName: "kube-api-access-rf996") pod "595fe665-fe7e-41f6-860d-280aa99be2fa" (UID: "595fe665-fe7e-41f6-860d-280aa99be2fa"). InnerVolumeSpecName "kube-api-access-rf996". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:04.881622 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.881594    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util" (OuterVolumeSpecName: "util") pod "595fe665-fe7e-41f6-860d-280aa99be2fa" (UID: "595fe665-fe7e-41f6-860d-280aa99be2fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:04.978804 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.978777    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-bundle\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:01:04.978804 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.978799    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf996\" (UniqueName: \"kubernetes.io/projected/595fe665-fe7e-41f6-860d-280aa99be2fa-kube-api-access-rf996\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:01:04.978937 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:04.978810    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/595fe665-fe7e-41f6-860d-280aa99be2fa-util\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:01:05.675222 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:05.675191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4"
Sep 17 16:01:05.675222 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:05.675196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a59kr4" event={"ID":"595fe665-fe7e-41f6-860d-280aa99be2fa","Type":"ContainerDied","Data":"7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b"}
Sep 17 16:01:05.675414 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:05.675259    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8d18ad52d37b4d932a6800ce2a618c01ffbb1e6f13ad5adc050650758a2e3b"
Sep 17 16:01:13.390589 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"]
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390830    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="util"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390842    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="util"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390875    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="extract"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390882    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="extract"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390889    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="pull"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="pull"
Sep 17 16:01:13.391041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.390939    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="595fe665-fe7e-41f6-860d-280aa99be2fa" containerName="extract"
Sep 17 16:01:13.394541 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.394523    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.397898 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.397876    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dm4mv\""
Sep 17 16:01:13.398433 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.398420    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:01:13.398510 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.398502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:01:13.398780 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.398764    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:01:13.423971 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.423948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"]
Sep 17 16:01:13.541977 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.541943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhxt\" (UniqueName: \"kubernetes.io/projected/cad012d9-aac0-45d8-80c9-361e5d24d2ed-kube-api-access-nbhxt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.542122 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.541988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad012d9-aac0-45d8-80c9-361e5d24d2ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.643382 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.643304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhxt\" (UniqueName: \"kubernetes.io/projected/cad012d9-aac0-45d8-80c9-361e5d24d2ed-kube-api-access-nbhxt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.643382 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.643336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad012d9-aac0-45d8-80c9-361e5d24d2ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.645600 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.645572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad012d9-aac0-45d8-80c9-361e5d24d2ed-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.657395 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.657371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhxt\" (UniqueName: \"kubernetes.io/projected/cad012d9-aac0-45d8-80c9-361e5d24d2ed-kube-api-access-nbhxt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-tzfbq\" (UID: \"cad012d9-aac0-45d8-80c9-361e5d24d2ed\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.703572 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.703551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:13.984155 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:13.984122    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"]
Sep 17 16:01:13.988275 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:01:13.988249    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad012d9_aac0_45d8_80c9_361e5d24d2ed.slice/crio-f5e28b344069659d65b19502b9dbd0df18a10f48809d7e65e5cdb510a8720ca3 WatchSource:0}: Error finding container f5e28b344069659d65b19502b9dbd0df18a10f48809d7e65e5cdb510a8720ca3: Status 404 returned error can't find the container with id f5e28b344069659d65b19502b9dbd0df18a10f48809d7e65e5cdb510a8720ca3
Sep 17 16:01:14.706551 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:14.706516    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq" event={"ID":"cad012d9-aac0-45d8-80c9-361e5d24d2ed","Type":"ContainerStarted","Data":"f5e28b344069659d65b19502b9dbd0df18a10f48809d7e65e5cdb510a8720ca3"}
Sep 17 16:01:18.723628 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:18.723573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq" event={"ID":"cad012d9-aac0-45d8-80c9-361e5d24d2ed","Type":"ContainerStarted","Data":"4fc51dcdea41d8d49bc0c3356e7f32a08bf1c6479effd5ab6cfb2794c37b9dee"}
Sep 17 16:01:18.724055 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:18.723733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:18.772404 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:18.772362    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq" podStartSLOduration=1.138611012 podStartE2EDuration="5.772350174s" totalImagesPullingTime="4.633739162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:18.770822799 +0000 UTC m=+329.626523362" watchObservedRunningTime="2026-09-17 16:01:18.772350174 +0000 UTC m=+329.628050733"
Sep 17 16:01:19.457719 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.457681    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kpmnb"]
Sep 17 16:01:19.460192 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.460176    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.464734 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.464712    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:01:19.465321 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.465301    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 16:01:19.465771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.465749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-r9s85\""
Sep 17 16:01:19.485402 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.485376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kpmnb"]
Sep 17 16:01:19.592726 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.592695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.592726 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.592726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/acd3081a-6667-487c-82aa-9dd30bc76456-cabundle0\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.592921 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.592826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qs5\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-kube-api-access-78qs5\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.693719 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.693692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.693719 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.693721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/acd3081a-6667-487c-82aa-9dd30bc76456-cabundle0\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.693913 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.693787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qs5\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-kube-api-access-78qs5\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.693913 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:19.693849    2562 secret.go:281] references non-existent secret key: ca.crt
Sep 17 16:01:19.693913 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:19.693870    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 16:01:19.693913 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:19.693879    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-kpmnb: references non-existent secret key: ca.crt
Sep 17 16:01:19.694065 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:19.693934    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates podName:acd3081a-6667-487c-82aa-9dd30bc76456 nodeName:}" failed. No retries permitted until 2026-09-17 16:01:20.193913565 +0000 UTC m=+331.049614104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates") pod "keda-operator-579c8958f7-kpmnb" (UID: "acd3081a-6667-487c-82aa-9dd30bc76456") : references non-existent secret key: ca.crt
Sep 17 16:01:19.694433 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.694415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/acd3081a-6667-487c-82aa-9dd30bc76456-cabundle0\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:19.709554 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:19.709492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qs5\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-kube-api-access-78qs5\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:20.198087 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.198058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:20.198531 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:20.198200    2562 secret.go:281] references non-existent secret key: ca.crt
Sep 17 16:01:20.198531 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:20.198221    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 16:01:20.198531 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:20.198234    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-kpmnb: references non-existent secret key: ca.crt
Sep 17 16:01:20.198531 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:01:20.198296    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates podName:acd3081a-6667-487c-82aa-9dd30bc76456 nodeName:}" failed. No retries permitted until 2026-09-17 16:01:21.198278681 +0000 UTC m=+332.053979220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates") pod "keda-operator-579c8958f7-kpmnb" (UID: "acd3081a-6667-487c-82aa-9dd30bc76456") : references non-existent secret key: ca.crt
Sep 17 16:01:20.300771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.300739    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"]
Sep 17 16:01:20.303225 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.303204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.308218 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.308197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 16:01:20.324897 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.324876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"]
Sep 17 16:01:20.400126 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.400096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.400126 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.400130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f810e651-99d5-4ea7-8f4a-fe0525d8580f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.400277 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.400151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4zb\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-kube-api-access-jj4zb\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.501371 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.501294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.501371 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.501329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f810e651-99d5-4ea7-8f4a-fe0525d8580f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.501371 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.501350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4zb\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-kube-api-access-jj4zb\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.501844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.501827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f810e651-99d5-4ea7-8f4a-fe0525d8580f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.503811 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.503786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.515413 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.515394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4zb\" (UniqueName: \"kubernetes.io/projected/f810e651-99d5-4ea7-8f4a-fe0525d8580f-kube-api-access-jj4zb\") pod \"keda-metrics-apiserver-77857b7b98-2ndrk\" (UID: \"f810e651-99d5-4ea7-8f4a-fe0525d8580f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.541609 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.541581    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jrj6d"]
Sep 17 16:01:20.543718 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.543694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.547485 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.547468    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 16:01:20.558860 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.558842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jrj6d"]
Sep 17 16:01:20.613508 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.613477    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:20.703558 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.703530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cjc\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-kube-api-access-t4cjc\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.703732 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.703600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-certificates\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.804564 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.804474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-certificates\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.804564 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.804521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cjc\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-kube-api-access-t4cjc\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.806931 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.806912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-certificates\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.817159 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.817128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cjc\" (UniqueName: \"kubernetes.io/projected/7d6c888d-0943-4752-afc5-08b40629a091-kube-api-access-t4cjc\") pod \"keda-admission-6c4d864b86-jrj6d\" (UID: \"7d6c888d-0943-4752-afc5-08b40629a091\") " pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.853797 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.853775    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:20.901119 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:20.901066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"]
Sep 17 16:01:20.906520 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:01:20.906456    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf810e651_99d5_4ea7_8f4a_fe0525d8580f.slice/crio-375333e15f4af870716b83b6b4d79f34e8e1efe4398e12a17f7f194f88c82bc6 WatchSource:0}: Error finding container 375333e15f4af870716b83b6b4d79f34e8e1efe4398e12a17f7f194f88c82bc6: Status 404 returned error can't find the container with id 375333e15f4af870716b83b6b4d79f34e8e1efe4398e12a17f7f194f88c82bc6
Sep 17 16:01:21.150321 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.150241    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jrj6d"]
Sep 17 16:01:21.153028 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:01:21.153007    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6c888d_0943_4752_afc5_08b40629a091.slice/crio-22e393168c2106acc1a993a364500e182c32ae15f8b026097570bf5a4a7e6a05 WatchSource:0}: Error finding container 22e393168c2106acc1a993a364500e182c32ae15f8b026097570bf5a4a7e6a05: Status 404 returned error can't find the container with id 22e393168c2106acc1a993a364500e182c32ae15f8b026097570bf5a4a7e6a05
Sep 17 16:01:21.208673 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.208643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:21.210985 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.210966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/acd3081a-6667-487c-82aa-9dd30bc76456-certificates\") pod \"keda-operator-579c8958f7-kpmnb\" (UID: \"acd3081a-6667-487c-82aa-9dd30bc76456\") " pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:21.270353 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.270320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:21.552149 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.552124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kpmnb"]
Sep 17 16:01:21.736037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.736000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d" event={"ID":"7d6c888d-0943-4752-afc5-08b40629a091","Type":"ContainerStarted","Data":"22e393168c2106acc1a993a364500e182c32ae15f8b026097570bf5a4a7e6a05"}
Sep 17 16:01:21.737097 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.737074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-kpmnb" event={"ID":"acd3081a-6667-487c-82aa-9dd30bc76456","Type":"ContainerStarted","Data":"8fd3b45b6cf2fd1e0af3171e3e73eda0eb0ef4d8ecb00e80b0775adc54338871"}
Sep 17 16:01:21.738082 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:21.738060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk" event={"ID":"f810e651-99d5-4ea7-8f4a-fe0525d8580f","Type":"ContainerStarted","Data":"375333e15f4af870716b83b6b4d79f34e8e1efe4398e12a17f7f194f88c82bc6"}
Sep 17 16:01:23.750071 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:23.750023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d" event={"ID":"7d6c888d-0943-4752-afc5-08b40629a091","Type":"ContainerStarted","Data":"4e3e43296df2386d13300acec41f02dff59023e55c5f76fa8708b15285b01084"}
Sep 17 16:01:23.750594 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:23.750157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:23.771895 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:23.771849    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d" podStartSLOduration=2.009558229 podStartE2EDuration="3.771830265s" totalImagesPullingTime="1.762272036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:23.770709529 +0000 UTC m=+334.626410113" watchObservedRunningTime="2026-09-17 16:01:23.771830265 +0000 UTC m=+334.627530828"
Sep 17 16:01:25.760897 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.760856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-kpmnb" event={"ID":"acd3081a-6667-487c-82aa-9dd30bc76456","Type":"ContainerStarted","Data":"589cf3ba631f53adbc640c29d8a5b9237fc1d810fc0dc900f8bfc90bbdc9b969"}
Sep 17 16:01:25.761334 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.760967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:01:25.762648 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.762604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk" event={"ID":"f810e651-99d5-4ea7-8f4a-fe0525d8580f","Type":"ContainerStarted","Data":"0972e846c76c65d41e93c0c4d8964f1d14223c3e927422450d3bb6b9a3f7da8e"}
Sep 17 16:01:25.762777 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.762739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:25.788561 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.788515    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-kpmnb" podStartSLOduration=3.172243113 podStartE2EDuration="6.788499014s" totalImagesPullingTime="3.616255901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:25.788328659 +0000 UTC m=+336.644029224" watchObservedRunningTime="2026-09-17 16:01:25.788499014 +0000 UTC m=+336.644199577"
Sep 17 16:01:25.817602 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:25.817558    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk" podStartSLOduration=1.562794046 podStartE2EDuration="5.817543617s" totalImagesPullingTime="4.254749571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:25.817108208 +0000 UTC m=+336.672808789" watchObservedRunningTime="2026-09-17 16:01:25.817543617 +0000 UTC m=+336.673244207"
Sep 17 16:01:36.771989 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:36.771915    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2ndrk"
Sep 17 16:01:39.730572 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:39.730541    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-tzfbq"
Sep 17 16:01:44.757147 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:44.757116    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jrj6d"
Sep 17 16:01:46.769735 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:01:46.769706    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-kpmnb"
Sep 17 16:02:43.132361 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.132330    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c8b97f4-58lxx"]
Sep 17 16:02:43.134501 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.134485    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.153517 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.153498    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c8b97f4-58lxx"]
Sep 17 16:02:43.176449 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-service-ca\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176568 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzgf\" (UniqueName: \"kubernetes.io/projected/2edd7c14-4486-48ce-86d8-c5583cafc747-kube-api-access-ldzgf\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176568 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-oauth-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176568 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-console-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176743 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-oauth-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176743 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-trusted-ca-bundle\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.176743 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.176667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277351 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-oauth-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-console-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-oauth-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-trusted-ca-bundle\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-service-ca\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.277521 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.277497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzgf\" (UniqueName: \"kubernetes.io/projected/2edd7c14-4486-48ce-86d8-c5583cafc747-kube-api-access-ldzgf\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.278235 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.278204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-oauth-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.278334 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.278206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-service-ca\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.278334 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.278206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-console-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.278479 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.278458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2edd7c14-4486-48ce-86d8-c5583cafc747-trusted-ca-bundle\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.280431 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.280412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-oauth-config\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.280431 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.280423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2edd7c14-4486-48ce-86d8-c5583cafc747-console-serving-cert\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.287996 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.287976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzgf\" (UniqueName: \"kubernetes.io/projected/2edd7c14-4486-48ce-86d8-c5583cafc747-kube-api-access-ldzgf\") pod \"console-85c8b97f4-58lxx\" (UID: \"2edd7c14-4486-48ce-86d8-c5583cafc747\") " pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.443479 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.443445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:43.719553 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:43.719526    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c8b97f4-58lxx"]
Sep 17 16:02:43.722139 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:02:43.722108    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edd7c14_4486_48ce_86d8_c5583cafc747.slice/crio-9993e952ce4273ca5a8176b7ad03b46ba9dd372a11fe750e0e374840443c415b WatchSource:0}: Error finding container 9993e952ce4273ca5a8176b7ad03b46ba9dd372a11fe750e0e374840443c415b: Status 404 returned error can't find the container with id 9993e952ce4273ca5a8176b7ad03b46ba9dd372a11fe750e0e374840443c415b
Sep 17 16:02:44.096455 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:44.096369    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c8b97f4-58lxx" event={"ID":"2edd7c14-4486-48ce-86d8-c5583cafc747","Type":"ContainerStarted","Data":"b1006a67df388c5655cae41f2db4ed5f78a6cd8ebaf81f3f4b6f0a8e059dfbdd"}
Sep 17 16:02:44.096455 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:44.096416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c8b97f4-58lxx" event={"ID":"2edd7c14-4486-48ce-86d8-c5583cafc747","Type":"ContainerStarted","Data":"9993e952ce4273ca5a8176b7ad03b46ba9dd372a11fe750e0e374840443c415b"}
Sep 17 16:02:44.130249 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:44.126590    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c8b97f4-58lxx" podStartSLOduration=1.126564892 podStartE2EDuration="1.126564892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:44.123344537 +0000 UTC m=+414.979045097" watchObservedRunningTime="2026-09-17 16:02:44.126564892 +0000 UTC m=+414.982265468"
Sep 17 16:02:45.447482 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.447445    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:02:45.449954 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.449929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.450570 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.450553    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"]
Sep 17 16:02:45.452474 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.452460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.453031 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.453017    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-54bz4\""
Sep 17 16:02:45.453473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.453459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 16:02:45.453747 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.453734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:02:45.454043 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.454023    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:02:45.454445 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.454423    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 16:02:45.454826 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.454808    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7kpk9\""
Sep 17 16:02:45.469352 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.469330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"]
Sep 17 16:02:45.477490 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.477470    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:02:45.496945 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.496923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66hl\" (UniqueName: \"kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.497058 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.496974    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk564\" (UniqueName: \"kubernetes.io/projected/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-kube-api-access-zk564\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.497058 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.497013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.497058 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.497036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.597694 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.597669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66hl\" (UniqueName: \"kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.597827 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.597742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk564\" (UniqueName: \"kubernetes.io/projected/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-kube-api-access-zk564\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.597827 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.597774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.597827 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.597799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.597999 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:02:45.597888    2562 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 16:02:45.597999 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:02:45.597967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert podName:145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2 nodeName:}" failed. No retries permitted until 2026-09-17 16:02:46.097951646 +0000 UTC m=+416.953652189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert") pod "llmisvc-controller-manager-645f9c8fd6-x5qcq" (UID: "145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2") : secret "llmisvc-webhook-server-cert" not found
Sep 17 16:02:45.600212 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.600191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.613114 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.613092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk564\" (UniqueName: \"kubernetes.io/projected/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-kube-api-access-zk564\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:45.624730 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.624707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66hl\" (UniqueName: \"kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl\") pod \"kserve-controller-manager-6fdd8844c6-c7bc6\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") " pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:45.763054 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:45.762991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:46.040571 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.040503    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:02:46.043021 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:02:46.042998    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2bfa91_e4b6_4000_8eab_d6a08594866a.slice/crio-cddeb88d7ac408b21db25f52f36ee05d921c1d444cc5262431edf5c172603e83 WatchSource:0}: Error finding container cddeb88d7ac408b21db25f52f36ee05d921c1d444cc5262431edf5c172603e83: Status 404 returned error can't find the container with id cddeb88d7ac408b21db25f52f36ee05d921c1d444cc5262431edf5c172603e83
Sep 17 16:02:46.101609 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.101582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:46.104152 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.104128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-x5qcq\" (UID: \"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:46.106310 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.106288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" event={"ID":"8b2bfa91-e4b6-4000-8eab-d6a08594866a","Type":"ContainerStarted","Data":"cddeb88d7ac408b21db25f52f36ee05d921c1d444cc5262431edf5c172603e83"}
Sep 17 16:02:46.370184 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.370106    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:46.643159 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:46.642960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"]
Sep 17 16:02:46.645402 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:02:46.645376    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod145e3a7e_4b06_4a71_9cd4_14cb57f2cdd2.slice/crio-6ce1f645b5c3809e808f4b0463a7486deb1cda7b46ef8d54e6d5d11511b8a82c WatchSource:0}: Error finding container 6ce1f645b5c3809e808f4b0463a7486deb1cda7b46ef8d54e6d5d11511b8a82c: Status 404 returned error can't find the container with id 6ce1f645b5c3809e808f4b0463a7486deb1cda7b46ef8d54e6d5d11511b8a82c
Sep 17 16:02:47.112179 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:47.112144    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq" event={"ID":"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2","Type":"ContainerStarted","Data":"6ce1f645b5c3809e808f4b0463a7486deb1cda7b46ef8d54e6d5d11511b8a82c"}
Sep 17 16:02:50.129926 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.129884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" event={"ID":"8b2bfa91-e4b6-4000-8eab-d6a08594866a","Type":"ContainerStarted","Data":"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"}
Sep 17 16:02:50.130367 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.130017    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:02:50.131397 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.131375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq" event={"ID":"145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2","Type":"ContainerStarted","Data":"197dbb14d5d1e2061e664840494360787b55431f135aef0be0eb08113fed507f"}
Sep 17 16:02:50.131516 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.131502    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:02:50.158106 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.158064    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" podStartSLOduration=1.683800507 podStartE2EDuration="5.15804934s" totalImagesPullingTime="3.474248833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:50.156456805 +0000 UTC m=+421.012157374" watchObservedRunningTime="2026-09-17 16:02:50.15804934 +0000 UTC m=+421.013749945"
Sep 17 16:02:50.185957 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:50.185916    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq" podStartSLOduration=2.312711211 podStartE2EDuration="5.185901501s" totalImagesPullingTime="2.87319029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:50.185580523 +0000 UTC m=+421.041281084" watchObservedRunningTime="2026-09-17 16:02:50.185901501 +0000 UTC m=+421.041602116"
Sep 17 16:02:53.444309 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:53.444262    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:53.444701 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:53.444357    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:53.449078 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:53.449045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:54.153256 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:54.153223    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c8b97f4-58lxx"
Sep 17 16:02:54.225942 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:02:54.225911    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 16:03:19.245030 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.244976    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5748cc9774-p8wg7" podUID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" containerName="console" containerID="cri-o://74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed" gracePeriod=15
Sep 17 16:03:19.487937 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.487914    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 16:03:19.571853 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.571853 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571859    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571914    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.571994    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvr9\" (UniqueName: \"kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572037 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert\") pod \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\" (UID: \"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd\") "
Sep 17 16:03:19.572284 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572129    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:19.572284 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572272    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-oauth-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.572392 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572296    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:19.572392 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:19.572392 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.572364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config" (OuterVolumeSpecName: "console-config") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:19.574149 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.574125    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:19.574149 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.574108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9" (OuterVolumeSpecName: "kube-api-access-wcvr9") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "kube-api-access-wcvr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:19.574317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.574186    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" (UID: "7b07fa26-a82b-48fb-8ea4-321d7aeef7bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:19.673357 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673322    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-service-ca\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.673357 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673348    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.673357 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673357    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvr9\" (UniqueName: \"kubernetes.io/projected/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-kube-api-access-wcvr9\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.673357 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673366    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-serving-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.673599 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673374    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-trusted-ca-bundle\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:19.673599 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:19.673383    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd-console-oauth-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:20.255786 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.255749    2562 generic.go:356] "Generic (PLEG): container finished" podID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" containerID="74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed" exitCode=0
Sep 17 16:03:20.256238 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.255797    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5748cc9774-p8wg7"
Sep 17 16:03:20.256238 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.255841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748cc9774-p8wg7" event={"ID":"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd","Type":"ContainerDied","Data":"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"}
Sep 17 16:03:20.256238 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.255907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5748cc9774-p8wg7" event={"ID":"7b07fa26-a82b-48fb-8ea4-321d7aeef7bd","Type":"ContainerDied","Data":"956e85c3ef1e0323f0ac83fe6b5141b37c3c9ba3e6289cdb382882a7c830e840"}
Sep 17 16:03:20.256238 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.255941    2562 scope.go:117] "RemoveContainer" containerID="74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"
Sep 17 16:03:20.264780 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.264762    2562 scope.go:117] "RemoveContainer" containerID="74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"
Sep 17 16:03:20.265034 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:03:20.265016    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed\": container with ID starting with 74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed not found: ID does not exist" containerID="74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"
Sep 17 16:03:20.265081 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.265043    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed"} err="failed to get container status \"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed\": rpc error: code = NotFound desc = could not find container \"74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed\": container with ID starting with 74436c6cdb7e5caac7ac0673d13f136fa91bca353239f4b93edbadc67b05cbed not found: ID does not exist"
Sep 17 16:03:20.291292 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.291266    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 16:03:20.296914 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:20.296894    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5748cc9774-p8wg7"]
Sep 17 16:03:21.137665 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:21.137634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-x5qcq"
Sep 17 16:03:21.140996 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:21.140976    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:03:21.474726 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:21.474697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" path="/var/lib/kubelet/pods/7b07fa26-a82b-48fb-8ea4-321d7aeef7bd/volumes"
Sep 17 16:03:34.636027 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.635989    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:03:34.636422 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.636174    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" podUID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" containerName="manager" containerID="cri-o://8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f" gracePeriod=10
Sep 17 16:03:34.711648 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.711600    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-mctfb"]
Sep 17 16:03:34.711987 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.711970    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" containerName="console"
Sep 17 16:03:34.712065 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.711990    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" containerName="console"
Sep 17 16:03:34.712122 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.712093    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b07fa26-a82b-48fb-8ea4-321d7aeef7bd" containerName="console"
Sep 17 16:03:34.714879 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.714860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.752359 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.752323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-mctfb"]
Sep 17 16:03:34.879226 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.879204    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:03:34.889242 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.889179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgk2\" (UniqueName: \"kubernetes.io/projected/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-kube-api-access-pcgk2\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.889344 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.889303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-cert\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.989926 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.989892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v66hl\" (UniqueName: \"kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl\") pod \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") "
Sep 17 16:03:34.990110 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.989931    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert\") pod \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\" (UID: \"8b2bfa91-e4b6-4000-8eab-d6a08594866a\") "
Sep 17 16:03:34.990110 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.990009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-cert\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.990110 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.990056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgk2\" (UniqueName: \"kubernetes.io/projected/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-kube-api-access-pcgk2\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.992184 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.992155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert" (OuterVolumeSpecName: "cert") pod "8b2bfa91-e4b6-4000-8eab-d6a08594866a" (UID: "8b2bfa91-e4b6-4000-8eab-d6a08594866a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:34.992388 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.992360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-cert\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:34.992501 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:34.992359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl" (OuterVolumeSpecName: "kube-api-access-v66hl") pod "8b2bfa91-e4b6-4000-8eab-d6a08594866a" (UID: "8b2bfa91-e4b6-4000-8eab-d6a08594866a"). InnerVolumeSpecName "kube-api-access-v66hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:35.006376 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.006352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgk2\" (UniqueName: \"kubernetes.io/projected/fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63-kube-api-access-pcgk2\") pod \"kserve-controller-manager-6fdd8844c6-mctfb\" (UID: \"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:35.024698 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.024672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:35.090810 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.090779    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66hl\" (UniqueName: \"kubernetes.io/projected/8b2bfa91-e4b6-4000-8eab-d6a08594866a-kube-api-access-v66hl\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:35.090810 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.090807    2562 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b2bfa91-e4b6-4000-8eab-d6a08594866a-cert\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:03:35.324520 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.324480    2562 generic.go:356] "Generic (PLEG): container finished" podID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" containerID="8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f" exitCode=0
Sep 17 16:03:35.324727 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.324537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" event={"ID":"8b2bfa91-e4b6-4000-8eab-d6a08594866a","Type":"ContainerDied","Data":"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"}
Sep 17 16:03:35.324727 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.324546    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6"
Sep 17 16:03:35.324727 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.324599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-c7bc6" event={"ID":"8b2bfa91-e4b6-4000-8eab-d6a08594866a","Type":"ContainerDied","Data":"cddeb88d7ac408b21db25f52f36ee05d921c1d444cc5262431edf5c172603e83"}
Sep 17 16:03:35.324727 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.324640    2562 scope.go:117] "RemoveContainer" containerID="8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"
Sep 17 16:03:35.329499 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.329477    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-mctfb"]
Sep 17 16:03:35.331381 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:03:35.331352    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6952ea_8c6c_449d_91f6_2e4b5f1c0e63.slice/crio-16a96105f89c0f33b16b9fc869d161a82226ba1952fa668b0bca87a64c51e629 WatchSource:0}: Error finding container 16a96105f89c0f33b16b9fc869d161a82226ba1952fa668b0bca87a64c51e629: Status 404 returned error can't find the container with id 16a96105f89c0f33b16b9fc869d161a82226ba1952fa668b0bca87a64c51e629
Sep 17 16:03:35.333707 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.333689    2562 scope.go:117] "RemoveContainer" containerID="8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"
Sep 17 16:03:35.334006 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:03:35.333970    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f\": container with ID starting with 8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f not found: ID does not exist" containerID="8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"
Sep 17 16:03:35.334067 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.334006    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f"} err="failed to get container status \"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f\": rpc error: code = NotFound desc = could not find container \"8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f\": container with ID starting with 8ab2bdfa1d9e5c4f95c0ec7c2030622f0c83840baf86b4953606bdab154df05f not found: ID does not exist"
Sep 17 16:03:35.383630 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.383591    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:03:35.384870 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.384848    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-c7bc6"]
Sep 17 16:03:35.474410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:35.474382    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" path="/var/lib/kubelet/pods/8b2bfa91-e4b6-4000-8eab-d6a08594866a/volumes"
Sep 17 16:03:36.330085 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:36.330052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb" event={"ID":"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63","Type":"ContainerStarted","Data":"8109cb4305de54e472e2354cec834dae37b19f7494f3c1eadf58bbd0b2e389c4"}
Sep 17 16:03:36.330522 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:36.330111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb" event={"ID":"fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63","Type":"ContainerStarted","Data":"16a96105f89c0f33b16b9fc869d161a82226ba1952fa668b0bca87a64c51e629"}
Sep 17 16:03:36.330522 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:36.330159    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:03:36.363581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:03:36.363541    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb" podStartSLOduration=1.999937553 podStartE2EDuration="2.363529389s" totalImagesPullingTime="363.591836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:36.361302847 +0000 UTC m=+467.217003418" watchObservedRunningTime="2026-09-17 16:03:36.363529389 +0000 UTC m=+467.219229950"
Sep 17 16:04:07.339444 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:07.339407    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-mctfb"
Sep 17 16:04:08.784513 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.784479    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-sph6m"]
Sep 17 16:04:08.784973 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.784832    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" containerName="manager"
Sep 17 16:04:08.784973 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.784845    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" containerName="manager"
Sep 17 16:04:08.784973 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.784906    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b2bfa91-e4b6-4000-8eab-d6a08594866a" containerName="manager"
Sep 17 16:04:08.787885 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.787859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.791006 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.790986    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-pg6m7\""
Sep 17 16:04:08.791227 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.791212    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 16:04:08.797851 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.797827    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-7dc6c"]
Sep 17 16:04:08.801277 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.801261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.805445 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.805425    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qzkvl\""
Sep 17 16:04:08.805756 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.805742    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:04:08.809130 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.809111    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:04:08.812608 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.812587    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sph6m"]
Sep 17 16:04:08.825055 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.824997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-7dc6c"]
Sep 17 16:04:08.836576 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.836545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.836693 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.836593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158921e5-4d81-4bd1-8754-bba6441f48ec-tls-certs\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.836693 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.836677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtnv\" (UniqueName: \"kubernetes.io/projected/90cc5f29-e6c9-4320-b52b-0184cbfc8605-kube-api-access-6gtnv\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.836801 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.836713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.836801 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.836786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w6h\" (UniqueName: \"kubernetes.io/projected/158921e5-4d81-4bd1-8754-bba6441f48ec-kube-api-access-w7w6h\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.937186 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.937159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w6h\" (UniqueName: \"kubernetes.io/projected/158921e5-4d81-4bd1-8754-bba6441f48ec-kube-api-access-w7w6h\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.937351 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.937195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.937351 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.937224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158921e5-4d81-4bd1-8754-bba6441f48ec-tls-certs\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.937351 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.937275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtnv\" (UniqueName: \"kubernetes.io/projected/90cc5f29-e6c9-4320-b52b-0184cbfc8605-kube-api-access-6gtnv\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.937351 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.937304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:08.937559 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:04:08.937353    2562 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:08.937559 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:04:08.937425    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert podName:90cc5f29-e6c9-4320-b52b-0184cbfc8605 nodeName:}" failed. No retries permitted until 2026-09-17 16:04:09.43740678 +0000 UTC m=+500.293107326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert") pod "odh-model-controller-7b966d7984-7dc6c" (UID: "90cc5f29-e6c9-4320-b52b-0184cbfc8605") : secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:08.937559 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:04:08.937439    2562 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 16:04:08.937559 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:04:08.937499    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs podName:90cc5f29-e6c9-4320-b52b-0184cbfc8605 nodeName:}" failed. No retries permitted until 2026-09-17 16:04:09.437482738 +0000 UTC m=+500.293183281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs") pod "odh-model-controller-7b966d7984-7dc6c" (UID: "90cc5f29-e6c9-4320-b52b-0184cbfc8605") : secret "odh-model-controller-metrics-tls" not found
Sep 17 16:04:08.939777 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.939759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158921e5-4d81-4bd1-8754-bba6441f48ec-tls-certs\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.949872 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.949842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w6h\" (UniqueName: \"kubernetes.io/projected/158921e5-4d81-4bd1-8754-bba6441f48ec-kube-api-access-w7w6h\") pod \"model-serving-api-7b954ffc95-sph6m\" (UID: \"158921e5-4d81-4bd1-8754-bba6441f48ec\") " pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:08.958893 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:08.958873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtnv\" (UniqueName: \"kubernetes.io/projected/90cc5f29-e6c9-4320-b52b-0184cbfc8605-kube-api-access-6gtnv\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.100285 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.100197    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:09.406844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.406766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-sph6m"]
Sep 17 16:04:09.409706 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:04:09.409681    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158921e5_4d81_4bd1_8754_bba6441f48ec.slice/crio-4e118b06d02c489383139d1da75210c223d949ab6a3ae55419b253e977019302 WatchSource:0}: Error finding container 4e118b06d02c489383139d1da75210c223d949ab6a3ae55419b253e977019302: Status 404 returned error can't find the container with id 4e118b06d02c489383139d1da75210c223d949ab6a3ae55419b253e977019302
Sep 17 16:04:09.442058 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.442035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.442148 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.442085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.444427 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.444400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-cert\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.444566 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.444548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc5f29-e6c9-4320-b52b-0184cbfc8605-metrics-certs\") pod \"odh-model-controller-7b966d7984-7dc6c\" (UID: \"90cc5f29-e6c9-4320-b52b-0184cbfc8605\") " pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.473787 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.473763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sph6m" event={"ID":"158921e5-4d81-4bd1-8754-bba6441f48ec","Type":"ContainerStarted","Data":"4e118b06d02c489383139d1da75210c223d949ab6a3ae55419b253e977019302"}
Sep 17 16:04:09.710773 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.710727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:09.989171 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:09.989092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-7dc6c"]
Sep 17 16:04:09.991989 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:04:09.991961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cc5f29_e6c9_4320_b52b_0184cbfc8605.slice/crio-258d2103b7d729ec67841941f74c9588e35e6d91e90a047fa184e2dd2e771b14 WatchSource:0}: Error finding container 258d2103b7d729ec67841941f74c9588e35e6d91e90a047fa184e2dd2e771b14: Status 404 returned error can't find the container with id 258d2103b7d729ec67841941f74c9588e35e6d91e90a047fa184e2dd2e771b14
Sep 17 16:04:10.478015 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:10.477978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-7dc6c" event={"ID":"90cc5f29-e6c9-4320-b52b-0184cbfc8605","Type":"ContainerStarted","Data":"258d2103b7d729ec67841941f74c9588e35e6d91e90a047fa184e2dd2e771b14"}
Sep 17 16:04:13.495593 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.495541    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-sph6m" event={"ID":"158921e5-4d81-4bd1-8754-bba6441f48ec","Type":"ContainerStarted","Data":"5f33d73993bb2d2bfb43319fe8079bd355c2e19850ca6a6d452f89399c51f368"}
Sep 17 16:04:13.496051 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.495659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:04:13.497149 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.497119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-7dc6c" event={"ID":"90cc5f29-e6c9-4320-b52b-0184cbfc8605","Type":"ContainerStarted","Data":"0d35c5768f0a07560df9cfad8397d1a653984aeb6dd20820800ab49142018b1a"}
Sep 17 16:04:13.497265 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.497177    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:13.535407 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.535356    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-sph6m" podStartSLOduration=1.791667757 podStartE2EDuration="5.535341809s" totalImagesPullingTime="3.743674052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:13.535130496 +0000 UTC m=+504.390831069" watchObservedRunningTime="2026-09-17 16:04:13.535341809 +0000 UTC m=+504.391042398"
Sep 17 16:04:13.600560 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:13.600505    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-7dc6c" podStartSLOduration=2.391745104 podStartE2EDuration="5.600490098s" totalImagesPullingTime="3.208744994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:13.600158934 +0000 UTC m=+504.455859521" watchObservedRunningTime="2026-09-17 16:04:13.600490098 +0000 UTC m=+504.456190658"
Sep 17 16:04:24.503570 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:24.503537    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-7dc6c"
Sep 17 16:04:24.505516 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:04:24.505496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-sph6m"
Sep 17 16:05:52.415244 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:52.415132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:05:52.415244 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:52.415132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:05:58.385072 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.385037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:05:58.387451 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.387434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.392791 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.392768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:05:58.393169 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.393153    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48b6f-predictor-serving-cert\""
Sep 17 16:05:58.393974 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.393946    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48b6f-kube-rbac-proxy-sar-config\""
Sep 17 16:05:58.397937 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.397922    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wx97n\""
Sep 17 16:05:58.406566 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.406544    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:05:58.408728 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.408701    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:05:58.439628 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.439593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.439781 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.439727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.439833 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.439779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t45c\" (UniqueName: \"kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.540482 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.540449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t45c\" (UniqueName: \"kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.540482 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.540485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.540707 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.540542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.540707 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:05:58.540675    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-serving-cert: secret "error-404-isvc-48b6f-predictor-serving-cert" not found
Sep 17 16:05:58.540800 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:05:58.540749    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls podName:fc0a8ab5-777e-492a-be25-6689499eb0e5 nodeName:}" failed. No retries permitted until 2026-09-17 16:05:59.040717928 +0000 UTC m=+609.896418467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls") pod "error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" (UID: "fc0a8ab5-777e-492a-be25-6689499eb0e5") : secret "error-404-isvc-48b6f-predictor-serving-cert" not found
Sep 17 16:05:58.541240 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.541220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:58.553318 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:58.553282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t45c\" (UniqueName: \"kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:59.046567 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.046536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:59.048909 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.048886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") pod \"error-404-isvc-48b6f-predictor-98c8c6df-zv5fb\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") " pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:59.300333 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.300239    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:05:59.587312 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.587229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:05:59.592735 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:05:59.592706    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a8ab5_777e_492a_be25_6689499eb0e5.slice/crio-5c9a5f31bf4dadf29444325a92dda37727b8091d2bdc03ca4938acd52d74bbf4 WatchSource:0}: Error finding container 5c9a5f31bf4dadf29444325a92dda37727b8091d2bdc03ca4938acd52d74bbf4: Status 404 returned error can't find the container with id 5c9a5f31bf4dadf29444325a92dda37727b8091d2bdc03ca4938acd52d74bbf4
Sep 17 16:05:59.594732 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.594711    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:05:59.947523 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:05:59.947493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerStarted","Data":"5c9a5f31bf4dadf29444325a92dda37727b8091d2bdc03ca4938acd52d74bbf4"}
Sep 17 16:06:12.012970 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:12.012933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerStarted","Data":"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"}
Sep 17 16:06:15.037683 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:15.037636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerStarted","Data":"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"}
Sep 17 16:06:15.038041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:15.037753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:06:15.039158 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:15.039128    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:15.069542 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:15.069496    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podStartSLOduration=2.762066261 podStartE2EDuration="17.069479209s" totalImagesPullingTime="14.307412948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:15.069404396 +0000 UTC m=+625.925104979" watchObservedRunningTime="2026-09-17 16:06:15.069479209 +0000 UTC m=+625.925179832"
Sep 17 16:06:16.042004 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:16.041968    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:16.042436 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:16.041978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:06:17.046032 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:17.045988    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:22.050470 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:22.050430    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:06:22.051019 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:22.050989    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:32.051035 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:32.050994    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:42.051518 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:42.051478    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:06:52.051195 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:06:52.051150    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 16:07:02.051927 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:02.051898    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:07:48.001718 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.001639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:07:48.002211 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.001910    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container" containerID="cri-o://8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4" gracePeriod=30
Sep 17 16:07:48.002211 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.001915    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kube-rbac-proxy" containerID="cri-o://82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b" gracePeriod=30
Sep 17 16:07:48.237201 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.237166    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:07:48.239775 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.239758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.242051 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.242031    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dabd-predictor-serving-cert\""
Sep 17 16:07:48.242165 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.242116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0dabd-kube-rbac-proxy-sar-config\""
Sep 17 16:07:48.250899 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.250879    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:07:48.362979 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.362900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.362979 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.362964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgcr\" (UniqueName: \"kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.363174 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.363017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.440661 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.440610    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerID="82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b" exitCode=2
Sep 17 16:07:48.440818 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.440685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerDied","Data":"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"}
Sep 17 16:07:48.464038 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.464001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.464191 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.464087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.464191 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.464133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgcr\" (UniqueName: \"kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.464378 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:07:48.464236    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-serving-cert: secret "error-404-isvc-0dabd-predictor-serving-cert" not found
Sep 17 16:07:48.464378 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:07:48.464290    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls podName:dd320469-feb2-4995-a991-be9db95c2b4a nodeName:}" failed. No retries permitted until 2026-09-17 16:07:48.964274325 +0000 UTC m=+719.819974864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls") pod "error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" (UID: "dd320469-feb2-4995-a991-be9db95c2b4a") : secret "error-404-isvc-0dabd-predictor-serving-cert" not found
Sep 17 16:07:48.464724 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.464707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.475238 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.475216    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgcr\" (UniqueName: \"kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.967449 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.967413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:48.969756 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:48.969725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") pod \"error-404-isvc-0dabd-predictor-7c8986597f-vl6mw\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") " pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:49.151330 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:49.151285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:49.438920 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:49.438844    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:07:49.443479 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:07:49.443446    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd320469_feb2_4995_a991_be9db95c2b4a.slice/crio-50b267d031aabafb6d29881cd97dccaa0319b36beda71a0a8d76d45cddb8de32 WatchSource:0}: Error finding container 50b267d031aabafb6d29881cd97dccaa0319b36beda71a0a8d76d45cddb8de32: Status 404 returned error can't find the container with id 50b267d031aabafb6d29881cd97dccaa0319b36beda71a0a8d76d45cddb8de32
Sep 17 16:07:50.450982 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.450941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerStarted","Data":"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"}
Sep 17 16:07:50.451346 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.451007    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:50.451346 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.451040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:50.451346 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.451064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerStarted","Data":"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"}
Sep 17 16:07:50.451346 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.451093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerStarted","Data":"50b267d031aabafb6d29881cd97dccaa0319b36beda71a0a8d76d45cddb8de32"}
Sep 17 16:07:50.452453 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.452428    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:07:50.486596 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:50.486550    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podStartSLOduration=2.486535842 podStartE2EDuration="2.486535842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:50.485373944 +0000 UTC m=+721.341074532" watchObservedRunningTime="2026-09-17 16:07:50.486535842 +0000 UTC m=+721.342236402"
Sep 17 16:07:51.154536 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.154505    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:07:51.288663 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.288566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config\") pod \"fc0a8ab5-777e-492a-be25-6689499eb0e5\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") "
Sep 17 16:07:51.288802 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.288687    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") pod \"fc0a8ab5-777e-492a-be25-6689499eb0e5\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") "
Sep 17 16:07:51.288802 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.288717    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t45c\" (UniqueName: \"kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c\") pod \"fc0a8ab5-777e-492a-be25-6689499eb0e5\" (UID: \"fc0a8ab5-777e-492a-be25-6689499eb0e5\") "
Sep 17 16:07:51.288938 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.288914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc0a8ab5-777e-492a-be25-6689499eb0e5" (UID: "fc0a8ab5-777e-492a-be25-6689499eb0e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:51.291022 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.290990    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c" (OuterVolumeSpecName: "kube-api-access-4t45c") pod "fc0a8ab5-777e-492a-be25-6689499eb0e5" (UID: "fc0a8ab5-777e-492a-be25-6689499eb0e5"). InnerVolumeSpecName "kube-api-access-4t45c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:51.291022 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.290998    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc0a8ab5-777e-492a-be25-6689499eb0e5" (UID: "fc0a8ab5-777e-492a-be25-6689499eb0e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:51.389672 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.389607    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:07:51.389672 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.389671    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0a8ab5-777e-492a-be25-6689499eb0e5-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:07:51.389842 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.389688    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t45c\" (UniqueName: \"kubernetes.io/projected/fc0a8ab5-777e-492a-be25-6689499eb0e5-kube-api-access-4t45c\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:07:51.457327 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.457296    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerID="8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4" exitCode=0
Sep 17 16:07:51.457680 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.457354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"
Sep 17 16:07:51.457680 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.457376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerDied","Data":"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"}
Sep 17 16:07:51.457680 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.457461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb" event={"ID":"fc0a8ab5-777e-492a-be25-6689499eb0e5","Type":"ContainerDied","Data":"5c9a5f31bf4dadf29444325a92dda37727b8091d2bdc03ca4938acd52d74bbf4"}
Sep 17 16:07:51.457680 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.457487    2562 scope.go:117] "RemoveContainer" containerID="82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"
Sep 17 16:07:51.458109 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.458076    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:07:51.466096 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.466080    2562 scope.go:117] "RemoveContainer" containerID="8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"
Sep 17 16:07:51.473439 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.473412    2562 scope.go:117] "RemoveContainer" containerID="82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"
Sep 17 16:07:51.473708 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:07:51.473686    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b\": container with ID starting with 82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b not found: ID does not exist" containerID="82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"
Sep 17 16:07:51.473788 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.473716    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b"} err="failed to get container status \"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b\": rpc error: code = NotFound desc = could not find container \"82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b\": container with ID starting with 82375debcac9f5e9d364d5e6438b8b787d5f0b7e392c874714815cd67d43822b not found: ID does not exist"
Sep 17 16:07:51.473788 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.473733    2562 scope.go:117] "RemoveContainer" containerID="8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"
Sep 17 16:07:51.474012 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:07:51.473990    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4\": container with ID starting with 8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4 not found: ID does not exist" containerID="8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"
Sep 17 16:07:51.474067 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.474018    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4"} err="failed to get container status \"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4\": rpc error: code = NotFound desc = could not find container \"8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4\": container with ID starting with 8173ce724e46a0042674a4aa06c93c44493016a6e83883fd179d4c72c4300da4 not found: ID does not exist"
Sep 17 16:07:51.496095 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.496059    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:07:51.502325 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:51.502302    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48b6f-predictor-98c8c6df-zv5fb"]
Sep 17 16:07:53.474692 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:53.474661    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" path="/var/lib/kubelet/pods/fc0a8ab5-777e-492a-be25-6689499eb0e5/volumes"
Sep 17 16:07:56.462014 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:56.461983    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:07:56.462476 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:07:56.462444    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:08:06.463227 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:06.463185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:08:16.462594 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:16.462550    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:08:26.462462 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:26.462415    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:08:29.245214 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245177    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:08:29.245581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245529    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kube-rbac-proxy"
Sep 17 16:08:29.245581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245541    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kube-rbac-proxy"
Sep 17 16:08:29.245581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245573    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container"
Sep 17 16:08:29.245581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container"
Sep 17 16:08:29.245738 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245654    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kserve-container"
Sep 17 16:08:29.245738 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.245666    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0a8ab5-777e-492a-be25-6689499eb0e5" containerName="kube-rbac-proxy"
Sep 17 16:08:29.247942 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.247924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.256861 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.256834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f67a-predictor-serving-cert\""
Sep 17 16:08:29.256983 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.256868    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f67a-kube-rbac-proxy-sar-config\""
Sep 17 16:08:29.281213 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.281184    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:08:29.304247 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.304212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sv2s\" (UniqueName: \"kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.304414 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.304322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.304414 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.304357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.405682 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.405636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.405844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.405698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sv2s\" (UniqueName: \"kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.405844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.405764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.406313 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.406286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.408150 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.408128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.419233 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.419213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sv2s\" (UniqueName: \"kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s\") pod \"error-404-isvc-4f67a-predictor-5476476989-b6s8s\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") " pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.558575 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.558493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:29.842689 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:29.842661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:08:29.844332 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:08:29.844308    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2182e6a_6b5b_49aa_a591_1f55b4abdd8c.slice/crio-118988da972917b5a27ceedd8cb09dab557e556c41d9db8bce185280c7822a0f WatchSource:0}: Error finding container 118988da972917b5a27ceedd8cb09dab557e556c41d9db8bce185280c7822a0f: Status 404 returned error can't find the container with id 118988da972917b5a27ceedd8cb09dab557e556c41d9db8bce185280c7822a0f
Sep 17 16:08:30.625966 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.625930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerStarted","Data":"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"}
Sep 17 16:08:30.626410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.625992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerStarted","Data":"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"}
Sep 17 16:08:30.626410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.626022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerStarted","Data":"118988da972917b5a27ceedd8cb09dab557e556c41d9db8bce185280c7822a0f"}
Sep 17 16:08:30.626410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.626160    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:30.626410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.626205    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:30.627271 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:30.627245    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:08:31.630609 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:31.630570    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:08:36.463807 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:36.463767    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:08:36.499723 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:36.499669    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podStartSLOduration=7.499653187 podStartE2EDuration="7.499653187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:30.650527637 +0000 UTC m=+761.506228220" watchObservedRunningTime="2026-09-17 16:08:36.499653187 +0000 UTC m=+767.355353745"
Sep 17 16:08:36.635488 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:36.635463    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:08:36.635944 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:36.635913    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:08:46.635934 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:46.635898    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:08:56.636054 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:08:56.636016    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:09:06.636708 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:09:06.636606    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:09:16.637567 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:09:16.637535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:10:52.441929 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:10:52.441896    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:10:52.443887 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:10:52.443473    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:15:52.469202 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:15:52.469101    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:15:52.471870 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:15:52.471848    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:17:03.077993 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.077918    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:17:03.078457 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.078147    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container" containerID="cri-o://723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6" gracePeriod=30
Sep 17 16:17:03.078457 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.078187    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kube-rbac-proxy" containerID="cri-o://057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76" gracePeriod=30
Sep 17 16:17:03.160088 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.160052    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:17:03.163503 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.163482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.166274 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.166252    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c0d2-kube-rbac-proxy-sar-config\""
Sep 17 16:17:03.166385 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.166252    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c0d2-predictor-serving-cert\""
Sep 17 16:17:03.177502 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.177481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:17:03.247885 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.247850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.247885 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.247892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.248082 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.247969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql549\" (UniqueName: \"kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.349404 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.349317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.349404 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.349357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.349404 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.349393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql549\" (UniqueName: \"kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.349721 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:03.349492    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-serving-cert: secret "error-404-isvc-7c0d2-predictor-serving-cert" not found
Sep 17 16:17:03.349721 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:03.349576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls podName:725717d9-9089-4815-a226-4e44a3a09d88 nodeName:}" failed. No retries permitted until 2026-09-17 16:17:03.849554894 +0000 UTC m=+1274.705255441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls") pod "error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" (UID: "725717d9-9089-4815-a226-4e44a3a09d88") : secret "error-404-isvc-7c0d2-predictor-serving-cert" not found
Sep 17 16:17:03.350083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.350059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.362665 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.362639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql549\" (UniqueName: \"kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.853059 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.853026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.855350 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.855321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") pod \"error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") " pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:03.897814 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.897786    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd320469-feb2-4995-a991-be9db95c2b4a" containerID="057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76" exitCode=2
Sep 17 16:17:03.897930 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:03.897821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerDied","Data":"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"}
Sep 17 16:17:04.076201 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.076165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:04.359964 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.359925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:17:04.367846 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:17:04.367809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725717d9_9089_4815_a226_4e44a3a09d88.slice/crio-db52845aea0d68cdeff7795e2e89ce4348463278a39ecd54580f04e0f9ec4cd1 WatchSource:0}: Error finding container db52845aea0d68cdeff7795e2e89ce4348463278a39ecd54580f04e0f9ec4cd1: Status 404 returned error can't find the container with id db52845aea0d68cdeff7795e2e89ce4348463278a39ecd54580f04e0f9ec4cd1
Sep 17 16:17:04.904282 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.904245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerStarted","Data":"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"}
Sep 17 16:17:04.904478 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.904313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerStarted","Data":"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"}
Sep 17 16:17:04.904478 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.904338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerStarted","Data":"db52845aea0d68cdeff7795e2e89ce4348463278a39ecd54580f04e0f9ec4cd1"}
Sep 17 16:17:04.904478 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.904360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:04.904478 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.904389    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:04.905601 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.905573    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:04.944317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:04.944264    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podStartSLOduration=1.9442498929999998 podStartE2EDuration="1.944249893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:04.942867605 +0000 UTC m=+1275.798568181" watchObservedRunningTime="2026-09-17 16:17:04.944249893 +0000 UTC m=+1275.799950454"
Sep 17 16:17:05.908633 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:05.908558    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:06.227832 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.227807    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:17:06.271820 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.271796    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config\") pod \"dd320469-feb2-4995-a991-be9db95c2b4a\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") "
Sep 17 16:17:06.271957 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.271888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") pod \"dd320469-feb2-4995-a991-be9db95c2b4a\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") "
Sep 17 16:17:06.271957 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.271915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgcr\" (UniqueName: \"kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr\") pod \"dd320469-feb2-4995-a991-be9db95c2b4a\" (UID: \"dd320469-feb2-4995-a991-be9db95c2b4a\") "
Sep 17 16:17:06.272119 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.272096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd320469-feb2-4995-a991-be9db95c2b4a" (UID: "dd320469-feb2-4995-a991-be9db95c2b4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:06.272176 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.272166    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd320469-feb2-4995-a991-be9db95c2b4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.274001 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.273972    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr" (OuterVolumeSpecName: "kube-api-access-kfgcr") pod "dd320469-feb2-4995-a991-be9db95c2b4a" (UID: "dd320469-feb2-4995-a991-be9db95c2b4a"). InnerVolumeSpecName "kube-api-access-kfgcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:06.274001 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.273988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd320469-feb2-4995-a991-be9db95c2b4a" (UID: "dd320469-feb2-4995-a991-be9db95c2b4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:06.372844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.372814    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd320469-feb2-4995-a991-be9db95c2b4a-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.372844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.372840    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgcr\" (UniqueName: \"kubernetes.io/projected/dd320469-feb2-4995-a991-be9db95c2b4a-kube-api-access-kfgcr\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:06.914017 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.913984    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd320469-feb2-4995-a991-be9db95c2b4a" containerID="723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6" exitCode=0
Sep 17 16:17:06.914473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.914030    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"
Sep 17 16:17:06.914473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.914070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerDied","Data":"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"}
Sep 17 16:17:06.914473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.914137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw" event={"ID":"dd320469-feb2-4995-a991-be9db95c2b4a","Type":"ContainerDied","Data":"50b267d031aabafb6d29881cd97dccaa0319b36beda71a0a8d76d45cddb8de32"}
Sep 17 16:17:06.914473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.914168    2562 scope.go:117] "RemoveContainer" containerID="057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"
Sep 17 16:17:06.923565 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.923550    2562 scope.go:117] "RemoveContainer" containerID="723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"
Sep 17 16:17:06.930606 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.930591    2562 scope.go:117] "RemoveContainer" containerID="057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"
Sep 17 16:17:06.930890 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:06.930872    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76\": container with ID starting with 057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76 not found: ID does not exist" containerID="057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"
Sep 17 16:17:06.930940 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.930898    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76"} err="failed to get container status \"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76\": rpc error: code = NotFound desc = could not find container \"057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76\": container with ID starting with 057966e87fcf374681112fd45a40d3ce4a88974fa7db2f37c6e692800a4c5b76 not found: ID does not exist"
Sep 17 16:17:06.930940 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.930913    2562 scope.go:117] "RemoveContainer" containerID="723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"
Sep 17 16:17:06.931146 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:06.931130    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6\": container with ID starting with 723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6 not found: ID does not exist" containerID="723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"
Sep 17 16:17:06.931187 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.931149    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6"} err="failed to get container status \"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6\": rpc error: code = NotFound desc = could not find container \"723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6\": container with ID starting with 723bf79803974982a863033218faef5feb70e73935312f5765af35df474f1df6 not found: ID does not exist"
Sep 17 16:17:06.941686 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.941665    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:17:06.952595 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:06.952573    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0dabd-predictor-7c8986597f-vl6mw"]
Sep 17 16:17:07.476137 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:07.476102    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" path="/var/lib/kubelet/pods/dd320469-feb2-4995-a991-be9db95c2b4a/volumes"
Sep 17 16:17:10.913970 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:10.913945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:10.914441 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:10.914409    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:20.914745 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:20.914705    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:30.914501 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:30.914457    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:40.915316 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:40.915271    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 16:17:43.513680 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.513642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:17:43.514142 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.513919    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" containerID="cri-o://c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca" gracePeriod=30
Sep 17 16:17:43.514142 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.513971    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kube-rbac-proxy" containerID="cri-o://85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2" gracePeriod=30
Sep 17 16:17:43.738864 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.738818    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:17:43.739185 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739172    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kube-rbac-proxy"
Sep 17 16:17:43.739253 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739186    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kube-rbac-proxy"
Sep 17 16:17:43.739253 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739197    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container"
Sep 17 16:17:43.739253 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739203    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container"
Sep 17 16:17:43.739400 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739260    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kserve-container"
Sep 17 16:17:43.739400 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.739267    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd320469-feb2-4995-a991-be9db95c2b4a" containerName="kube-rbac-proxy"
Sep 17 16:17:43.742193 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.742168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.745334 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.745312    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48574-predictor-serving-cert\""
Sep 17 16:17:43.745467 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.745342    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-48574-kube-rbac-proxy-sar-config\""
Sep 17 16:17:43.754417 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.754395    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:17:43.880825 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.880745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.880825 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.880787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtn87\" (UniqueName: \"kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.881041 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.880828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.981888 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.981851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtn87\" (UniqueName: \"kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.982069 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.981943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.982069 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.982015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.982069 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:43.982043    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-48574-predictor-serving-cert: secret "error-404-isvc-48574-predictor-serving-cert" not found
Sep 17 16:17:43.982247 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:43.982138    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls podName:fc0b2309-ae47-4647-b7c9-dfc97cdc6aae nodeName:}" failed. No retries permitted until 2026-09-17 16:17:44.48211805 +0000 UTC m=+1315.337818594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls") pod "error-404-isvc-48574-predictor-768499d5df-spw9k" (UID: "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae") : secret "error-404-isvc-48574-predictor-serving-cert" not found
Sep 17 16:17:43.982757 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.982739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:43.995532 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:43.995504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtn87\" (UniqueName: \"kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:44.081980 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.081945    2562 generic.go:356] "Generic (PLEG): container finished" podID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerID="85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2" exitCode=2
Sep 17 16:17:44.082134 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.082017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerDied","Data":"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"}
Sep 17 16:17:44.484666 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.484611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:44.487016 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.486992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") pod \"error-404-isvc-48574-predictor-768499d5df-spw9k\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") " pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:44.654103 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.654070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:44.934554 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:44.934459    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:17:44.936777 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:17:44.936746    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0b2309_ae47_4647_b7c9_dfc97cdc6aae.slice/crio-65953a56bee40aa54796b21d54d15757b6c694dc80c6b3c3f65ff0644be995a1 WatchSource:0}: Error finding container 65953a56bee40aa54796b21d54d15757b6c694dc80c6b3c3f65ff0644be995a1: Status 404 returned error can't find the container with id 65953a56bee40aa54796b21d54d15757b6c694dc80c6b3c3f65ff0644be995a1
Sep 17 16:17:45.088478 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.088430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerStarted","Data":"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"}
Sep 17 16:17:45.088644 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.088500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerStarted","Data":"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"}
Sep 17 16:17:45.088644 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.088532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerStarted","Data":"65953a56bee40aa54796b21d54d15757b6c694dc80c6b3c3f65ff0644be995a1"}
Sep 17 16:17:45.088770 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.088678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:45.088770 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.088732    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:45.090479 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.090444    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:17:45.108726 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:45.108669    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podStartSLOduration=2.108653211 podStartE2EDuration="2.108653211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:45.107961993 +0000 UTC m=+1315.963662592" watchObservedRunningTime="2026-09-17 16:17:45.108653211 +0000 UTC m=+1315.964353771"
Sep 17 16:17:46.094562 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:46.094508    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:17:46.631335 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:46.631295    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 16:17:46.636826 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:46.636803    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:17:46.857650 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:46.857628    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:17:47.009418 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.009387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sv2s\" (UniqueName: \"kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s\") pod \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") "
Sep 17 16:17:47.009418 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.009418    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls\") pod \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") "
Sep 17 16:17:47.009671 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.009518    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config\") pod \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\" (UID: \"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c\") "
Sep 17 16:17:47.009907 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.009876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" (UID: "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:47.011377 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.011340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s" (OuterVolumeSpecName: "kube-api-access-4sv2s") pod "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" (UID: "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c"). InnerVolumeSpecName "kube-api-access-4sv2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:47.011466 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.011408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" (UID: "e2182e6a-6b5b-49aa-a591-1f55b4abdd8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:47.101393 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.101361    2562 generic.go:356] "Generic (PLEG): container finished" podID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerID="c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca" exitCode=0
Sep 17 16:17:47.101771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.101409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerDied","Data":"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"}
Sep 17 16:17:47.101771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.101409    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"
Sep 17 16:17:47.101771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.101486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s" event={"ID":"e2182e6a-6b5b-49aa-a591-1f55b4abdd8c","Type":"ContainerDied","Data":"118988da972917b5a27ceedd8cb09dab557e556c41d9db8bce185280c7822a0f"}
Sep 17 16:17:47.101771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.101514    2562 scope.go:117] "RemoveContainer" containerID="85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"
Sep 17 16:17:47.110013 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.109995    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:47.110013 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.110015    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sv2s\" (UniqueName: \"kubernetes.io/projected/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-kube-api-access-4sv2s\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:47.110134 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.110024    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:17:47.110289 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.110273    2562 scope.go:117] "RemoveContainer" containerID="c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"
Sep 17 16:17:47.117347 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.117328    2562 scope.go:117] "RemoveContainer" containerID="85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"
Sep 17 16:17:47.117585 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:47.117567    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2\": container with ID starting with 85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2 not found: ID does not exist" containerID="85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"
Sep 17 16:17:47.117663 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.117592    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2"} err="failed to get container status \"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2\": rpc error: code = NotFound desc = could not find container \"85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2\": container with ID starting with 85d3ec170949233e1f0260305ec0c3e37fa8a76e532ed99567ee0cb8ddd921c2 not found: ID does not exist"
Sep 17 16:17:47.117663 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.117607    2562 scope.go:117] "RemoveContainer" containerID="c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"
Sep 17 16:17:47.117843 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:17:47.117825    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca\": container with ID starting with c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca not found: ID does not exist" containerID="c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"
Sep 17 16:17:47.117883 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.117847    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca"} err="failed to get container status \"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca\": rpc error: code = NotFound desc = could not find container \"c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca\": container with ID starting with c62bb84413e97050615b5c950ad06988618e20ad3b8c1c69befadee07b1104ca not found: ID does not exist"
Sep 17 16:17:47.127270 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.127250    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:17:47.132362 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.132342    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f67a-predictor-5476476989-b6s8s"]
Sep 17 16:17:47.474643 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:47.474598    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" path="/var/lib/kubelet/pods/e2182e6a-6b5b-49aa-a591-1f55b4abdd8c/volumes"
Sep 17 16:17:50.915329 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:50.915300    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:17:51.098975 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:51.098945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:17:51.099374 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:17:51.099352    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:18:01.099566 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:01.099527    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:18:11.100358 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:11.100318    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:18:13.413859 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.413830    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:18:13.414277 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.414043    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container" containerID="cri-o://385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce" gracePeriod=30
Sep 17 16:18:13.414277 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.414095    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kube-rbac-proxy" containerID="cri-o://4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf" gracePeriod=30
Sep 17 16:18:13.780489 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780455    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:18:13.780836 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780823    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container"
Sep 17 16:18:13.780895 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780838    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container"
Sep 17 16:18:13.780895 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780854    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kube-rbac-proxy"
Sep 17 16:18:13.780895 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780860    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kube-rbac-proxy"
Sep 17 16:18:13.780987 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780914    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kube-rbac-proxy"
Sep 17 16:18:13.780987 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.780923    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2182e6a-6b5b-49aa-a591-1f55b4abdd8c" containerName="kserve-container"
Sep 17 16:18:13.783978 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.783962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.788686 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.788659    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e5cb-predictor-serving-cert\""
Sep 17 16:18:13.788809 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.788722    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1e5cb-kube-rbac-proxy-sar-config\""
Sep 17 16:18:13.815252 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.815223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.815393 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.815262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.815393 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.815298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7bft\" (UniqueName: \"kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.815555 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.815531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:18:13.916003 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.915969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bft\" (UniqueName: \"kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.916164 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.916044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.916164 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.916079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.916253 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:13.916184    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-serving-cert: secret "error-404-isvc-1e5cb-predictor-serving-cert" not found
Sep 17 16:18:13.916253 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:13.916252    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls podName:f82dd29d-21f3-459b-a036-700eca9352f2 nodeName:}" failed. No retries permitted until 2026-09-17 16:18:14.416234748 +0000 UTC m=+1345.271935301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls") pod "error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" (UID: "f82dd29d-21f3-459b-a036-700eca9352f2") : secret "error-404-isvc-1e5cb-predictor-serving-cert" not found
Sep 17 16:18:13.916726 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.916708    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:13.937639 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:13.937586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bft\" (UniqueName: \"kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:14.227901 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.227866    2562 generic.go:356] "Generic (PLEG): container finished" podID="725717d9-9089-4815-a226-4e44a3a09d88" containerID="4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf" exitCode=2
Sep 17 16:18:14.227901 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.227903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerDied","Data":"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"}
Sep 17 16:18:14.419504 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.419465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:14.421993 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.421960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") pod \"error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") " pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:14.696587 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.696552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:14.981738 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:14.981655    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:18:14.984575 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:18:14.984542    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82dd29d_21f3_459b_a036_700eca9352f2.slice/crio-0292d230a56707913ff0b7759e70d8e9d15aae8e49ae8e4ac916a06e3eb61a96 WatchSource:0}: Error finding container 0292d230a56707913ff0b7759e70d8e9d15aae8e49ae8e4ac916a06e3eb61a96: Status 404 returned error can't find the container with id 0292d230a56707913ff0b7759e70d8e9d15aae8e49ae8e4ac916a06e3eb61a96
Sep 17 16:18:15.234137 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.234048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerStarted","Data":"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"}
Sep 17 16:18:15.234137 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.234105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerStarted","Data":"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"}
Sep 17 16:18:15.234137 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.234126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerStarted","Data":"0292d230a56707913ff0b7759e70d8e9d15aae8e49ae8e4ac916a06e3eb61a96"}
Sep 17 16:18:15.234388 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.234158    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:15.234388 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.234298    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:15.235773 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.235751    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:15.270301 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.270251    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podStartSLOduration=2.270237833 podStartE2EDuration="2.270237833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:15.268086966 +0000 UTC m=+1346.123787547" watchObservedRunningTime="2026-09-17 16:18:15.270237833 +0000 UTC m=+1346.125938394"
Sep 17 16:18:15.909069 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:15.909021    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 16:18:16.238321 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.238282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:16.562709 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.562683    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:18:16.636850 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.636821    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config\") pod \"725717d9-9089-4815-a226-4e44a3a09d88\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") "
Sep 17 16:18:16.637030 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.636871    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") pod \"725717d9-9089-4815-a226-4e44a3a09d88\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") "
Sep 17 16:18:16.637030 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.636917    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql549\" (UniqueName: \"kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549\") pod \"725717d9-9089-4815-a226-4e44a3a09d88\" (UID: \"725717d9-9089-4815-a226-4e44a3a09d88\") "
Sep 17 16:18:16.637235 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.637208    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "725717d9-9089-4815-a226-4e44a3a09d88" (UID: "725717d9-9089-4815-a226-4e44a3a09d88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:16.638789 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.638769    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549" (OuterVolumeSpecName: "kube-api-access-ql549") pod "725717d9-9089-4815-a226-4e44a3a09d88" (UID: "725717d9-9089-4815-a226-4e44a3a09d88"). InnerVolumeSpecName "kube-api-access-ql549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:16.638879 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.638804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "725717d9-9089-4815-a226-4e44a3a09d88" (UID: "725717d9-9089-4815-a226-4e44a3a09d88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:16.737638 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.737593    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/725717d9-9089-4815-a226-4e44a3a09d88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:16.737638 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.737634    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/725717d9-9089-4815-a226-4e44a3a09d88-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:16.737638 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:16.737645    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql549\" (UniqueName: \"kubernetes.io/projected/725717d9-9089-4815-a226-4e44a3a09d88-kube-api-access-ql549\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:17.244336 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.244302    2562 generic.go:356] "Generic (PLEG): container finished" podID="725717d9-9089-4815-a226-4e44a3a09d88" containerID="385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce" exitCode=0
Sep 17 16:18:17.244336 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.244339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerDied","Data":"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"}
Sep 17 16:18:17.244830 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.244351    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"
Sep 17 16:18:17.244830 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.244411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9" event={"ID":"725717d9-9089-4815-a226-4e44a3a09d88","Type":"ContainerDied","Data":"db52845aea0d68cdeff7795e2e89ce4348463278a39ecd54580f04e0f9ec4cd1"}
Sep 17 16:18:17.244830 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.244440    2562 scope.go:117] "RemoveContainer" containerID="4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"
Sep 17 16:18:17.253016 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.253000    2562 scope.go:117] "RemoveContainer" containerID="385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"
Sep 17 16:18:17.260135 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.260117    2562 scope.go:117] "RemoveContainer" containerID="4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"
Sep 17 16:18:17.260385 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:17.260367    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf\": container with ID starting with 4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf not found: ID does not exist" containerID="4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"
Sep 17 16:18:17.260451 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.260395    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf"} err="failed to get container status \"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf\": rpc error: code = NotFound desc = could not find container \"4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf\": container with ID starting with 4856d0cd11d72e5b635ac01dbc09befe06fb5f54affee3e56019f1d1ebff0abf not found: ID does not exist"
Sep 17 16:18:17.260451 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.260418    2562 scope.go:117] "RemoveContainer" containerID="385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"
Sep 17 16:18:17.260658 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:17.260635    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce\": container with ID starting with 385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce not found: ID does not exist" containerID="385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"
Sep 17 16:18:17.260707 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.260664    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce"} err="failed to get container status \"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce\": rpc error: code = NotFound desc = could not find container \"385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce\": container with ID starting with 385dd97c70075ae3d3f4920578d3c7e9792188ca4ff21ef3bdfd68b87e2765ce not found: ID does not exist"
Sep 17 16:18:17.280986 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.280963    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:18:17.284781 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.284759    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c0d2-predictor-dbcf8947c-cd8k9"]
Sep 17 16:18:17.475292 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:17.475258    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725717d9-9089-4815-a226-4e44a3a09d88" path="/var/lib/kubelet/pods/725717d9-9089-4815-a226-4e44a3a09d88/volumes"
Sep 17 16:18:21.099476 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:21.099433    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:18:21.242515 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:21.242485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:18:21.242970 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:21.242940    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:31.100785 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:31.100755    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:18:31.243334 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:31.243297    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:41.243650 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:41.243592    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:51.243864 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:51.243822    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:18:53.997982 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:53.997949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:18:53.998410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:53.998236    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container" containerID="cri-o://fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54" gracePeriod=30
Sep 17 16:18:53.998410 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:53.998269    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kube-rbac-proxy" containerID="cri-o://4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7" gracePeriod=30
Sep 17 16:18:54.196931 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.196892    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:18:54.197422 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197401    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kube-rbac-proxy"
Sep 17 16:18:54.197465 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197427    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kube-rbac-proxy"
Sep 17 16:18:54.197465 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197441    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container"
Sep 17 16:18:54.197465 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197448    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container"
Sep 17 16:18:54.197558 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197536    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kube-rbac-proxy"
Sep 17 16:18:54.197558 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.197549    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="725717d9-9089-4815-a226-4e44a3a09d88" containerName="kserve-container"
Sep 17 16:18:54.200894 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.200877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.203520 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.203499    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-26f37-predictor-serving-cert\""
Sep 17 16:18:54.203654 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.203558    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-26f37-kube-rbac-proxy-sar-config\""
Sep 17 16:18:54.219186 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.219164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:18:54.358524 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.358439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzgw\" (UniqueName: \"kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.358524 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.358482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.358524 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.358506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.415996 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.415953    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerID="4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7" exitCode=2
Sep 17 16:18:54.416142 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.416053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerDied","Data":"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"}
Sep 17 16:18:54.459053 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.459017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzgw\" (UniqueName: \"kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.459053 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.459058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.459286 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.459083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.459286 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:54.459216    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-26f37-predictor-serving-cert: secret "error-404-isvc-26f37-predictor-serving-cert" not found
Sep 17 16:18:54.459286 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:54.459272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls podName:30952538-669f-4d74-960f-ae136009a7bc nodeName:}" failed. No retries permitted until 2026-09-17 16:18:54.959254354 +0000 UTC m=+1385.814954892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls") pod "error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" (UID: "30952538-669f-4d74-960f-ae136009a7bc") : secret "error-404-isvc-26f37-predictor-serving-cert" not found
Sep 17 16:18:54.459748 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.459728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.474258 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.474230    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzgw\" (UniqueName: \"kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.963050 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.963012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:54.965560 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:54.965539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") pod \"error-404-isvc-26f37-predictor-64fcbb98df-qdjgh\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") " pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:55.111489 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:55.111448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:55.404093 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:55.404065    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:18:55.406595 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:18:55.406564    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30952538_669f_4d74_960f_ae136009a7bc.slice/crio-85636c411c52315a5d098b02abb8c4def9d159b61bac09bf23a70ae360dd316f WatchSource:0}: Error finding container 85636c411c52315a5d098b02abb8c4def9d159b61bac09bf23a70ae360dd316f: Status 404 returned error can't find the container with id 85636c411c52315a5d098b02abb8c4def9d159b61bac09bf23a70ae360dd316f
Sep 17 16:18:55.423006 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:55.422972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerStarted","Data":"85636c411c52315a5d098b02abb8c4def9d159b61bac09bf23a70ae360dd316f"}
Sep 17 16:18:56.095447 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.095382    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 16:18:56.429123 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.429040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerStarted","Data":"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"}
Sep 17 16:18:56.429123 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.429101    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:56.429123 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.429121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerStarted","Data":"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"}
Sep 17 16:18:56.429544 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.429159    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:18:56.430789 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.430764    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:18:56.463802 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:56.463750    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podStartSLOduration=2.463736294 podStartE2EDuration="2.463736294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:56.462811975 +0000 UTC m=+1387.318512536" watchObservedRunningTime="2026-09-17 16:18:56.463736294 +0000 UTC m=+1387.319436854"
Sep 17 16:18:57.242109 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.242082    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:18:57.387095 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.387005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtn87\" (UniqueName: \"kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87\") pod \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") "
Sep 17 16:18:57.387095 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.387085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config\") pod \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") "
Sep 17 16:18:57.387284 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.387104    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") pod \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\" (UID: \"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae\") "
Sep 17 16:18:57.387454 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.387429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" (UID: "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:57.389285 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.389256    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" (UID: "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:57.389709 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.389686    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87" (OuterVolumeSpecName: "kube-api-access-jtn87") pod "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" (UID: "fc0b2309-ae47-4647-b7c9-dfc97cdc6aae"). InnerVolumeSpecName "kube-api-access-jtn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:57.435229 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435198    2562 generic.go:356] "Generic (PLEG): container finished" podID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerID="fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54" exitCode=0
Sep 17 16:18:57.435678 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435253    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"
Sep 17 16:18:57.435678 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerDied","Data":"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"}
Sep 17 16:18:57.435678 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k" event={"ID":"fc0b2309-ae47-4647-b7c9-dfc97cdc6aae","Type":"ContainerDied","Data":"65953a56bee40aa54796b21d54d15757b6c694dc80c6b3c3f65ff0644be995a1"}
Sep 17 16:18:57.435678 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435373    2562 scope.go:117] "RemoveContainer" containerID="4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"
Sep 17 16:18:57.435992 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.435958    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:18:57.444305 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.444290    2562 scope.go:117] "RemoveContainer" containerID="fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"
Sep 17 16:18:57.451734 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.451717    2562 scope.go:117] "RemoveContainer" containerID="4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"
Sep 17 16:18:57.451982 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:57.451964    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7\": container with ID starting with 4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7 not found: ID does not exist" containerID="4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"
Sep 17 16:18:57.452050 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.451994    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7"} err="failed to get container status \"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7\": rpc error: code = NotFound desc = could not find container \"4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7\": container with ID starting with 4d2dbd0684538847a5b13d4faba3aab5e44a632e578806638ae5629e320695d7 not found: ID does not exist"
Sep 17 16:18:57.452050 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.452017    2562 scope.go:117] "RemoveContainer" containerID="fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"
Sep 17 16:18:57.452280 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:18:57.452263    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54\": container with ID starting with fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54 not found: ID does not exist" containerID="fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"
Sep 17 16:18:57.452322 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.452285    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54"} err="failed to get container status \"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54\": rpc error: code = NotFound desc = could not find container \"fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54\": container with ID starting with fa4cd13349ea4e1cf71cf4bba2cb7791f22383668c45b1a6bf8e778ebd302e54 not found: ID does not exist"
Sep 17 16:18:57.476517 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.476493    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:18:57.476607 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.476523    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-48574-predictor-768499d5df-spw9k"]
Sep 17 16:18:57.487772 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.487750    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:57.487772 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.487772    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:57.487910 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:57.487787    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtn87\" (UniqueName: \"kubernetes.io/projected/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae-kube-api-access-jtn87\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:18:59.475171 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:18:59.475141    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" path="/var/lib/kubelet/pods/fc0b2309-ae47-4647-b7c9-dfc97cdc6aae/volumes"
Sep 17 16:19:01.244659 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:01.244605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:19:02.441512 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:02.441485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:19:02.442035 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:02.442009    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:19:12.442650 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:12.442591    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:19:22.442586 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:22.442542    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:19:32.442989 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:32.442941    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 16:19:42.443669 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:19:42.443641    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:20:52.499591 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:20:52.499479    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:20:52.502412 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:20:52.502386    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:25:52.525712 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:25:52.525592    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:25:52.531771 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:25:52.530049    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:27:28.718578 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.718542    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:27:28.719153 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.718779    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" containerID="cri-o://b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd" gracePeriod=30
Sep 17 16:27:28.719153 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.718830    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kube-rbac-proxy" containerID="cri-o://8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58" gracePeriod=30
Sep 17 16:27:28.953227 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953189    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:27:28.953581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953568    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kube-rbac-proxy"
Sep 17 16:27:28.953581 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kube-rbac-proxy"
Sep 17 16:27:28.953728 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953591    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container"
Sep 17 16:27:28.953728 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953596    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container"
Sep 17 16:27:28.953728 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953660    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kube-rbac-proxy"
Sep 17 16:27:28.953728 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.953666    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0b2309-ae47-4647-b7c9-dfc97cdc6aae" containerName="kserve-container"
Sep 17 16:27:28.956585 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.956566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:28.960067 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.960043    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ce9a-predictor-serving-cert\""
Sep 17 16:27:28.960303 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.960283    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7ce9a-kube-rbac-proxy-sar-config\""
Sep 17 16:27:28.969885 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:28.969832    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:27:29.036362 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.036332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjt2k\" (UniqueName: \"kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.036545 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.036381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.036545 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.036452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.137661 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.137597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjt2k\" (UniqueName: \"kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.137889 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.137707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.137889 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.137756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.137889 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:27:29.137867    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-serving-cert: secret "error-404-isvc-7ce9a-predictor-serving-cert" not found
Sep 17 16:27:29.138073 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:27:29.137942    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls podName:78af82a4-b903-4ebd-8908-57f189e04659 nodeName:}" failed. No retries permitted until 2026-09-17 16:27:29.637920784 +0000 UTC m=+1900.493621338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls") pod "error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" (UID: "78af82a4-b903-4ebd-8908-57f189e04659") : secret "error-404-isvc-7ce9a-predictor-serving-cert" not found
Sep 17 16:27:29.138459 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.138440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.149105 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.149083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjt2k\" (UniqueName: \"kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.642276 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.642235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.644662 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.644637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") pod \"error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") " pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:29.712994 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.712963    2562 generic.go:356] "Generic (PLEG): container finished" podID="f82dd29d-21f3-459b-a036-700eca9352f2" containerID="8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58" exitCode=2
Sep 17 16:27:29.713127 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.713025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerDied","Data":"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"}
Sep 17 16:27:29.866509 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:29.866456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:30.144790 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.144762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:27:30.147292 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:27:30.147264    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af82a4_b903_4ebd_8908_57f189e04659.slice/crio-1e995183677c42702ed65a4ce23d5ba017824c547df2f469dc5ef2529626171c WatchSource:0}: Error finding container 1e995183677c42702ed65a4ce23d5ba017824c547df2f469dc5ef2529626171c: Status 404 returned error can't find the container with id 1e995183677c42702ed65a4ce23d5ba017824c547df2f469dc5ef2529626171c
Sep 17 16:27:30.719991 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.719954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerStarted","Data":"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"}
Sep 17 16:27:30.720152 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.720021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerStarted","Data":"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"}
Sep 17 16:27:30.720152 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.720051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerStarted","Data":"1e995183677c42702ed65a4ce23d5ba017824c547df2f469dc5ef2529626171c"}
Sep 17 16:27:30.720152 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.720105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:30.745973 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:30.745930    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podStartSLOduration=2.745915988 podStartE2EDuration="2.745915988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:30.744391347 +0000 UTC m=+1901.600091924" watchObservedRunningTime="2026-09-17 16:27:30.745915988 +0000 UTC m=+1901.601616546"
Sep 17 16:27:31.239169 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:31.239124    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 16:27:31.243492 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:31.243469    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:27:31.724993 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:31.724957    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:31.726210 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:31.726178    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:27:31.978412 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:31.978357    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:27:32.061394 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.061362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config\") pod \"f82dd29d-21f3-459b-a036-700eca9352f2\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") "
Sep 17 16:27:32.061574 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.061423    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") pod \"f82dd29d-21f3-459b-a036-700eca9352f2\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") "
Sep 17 16:27:32.061574 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.061461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7bft\" (UniqueName: \"kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft\") pod \"f82dd29d-21f3-459b-a036-700eca9352f2\" (UID: \"f82dd29d-21f3-459b-a036-700eca9352f2\") "
Sep 17 16:27:32.061772 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.061746    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f82dd29d-21f3-459b-a036-700eca9352f2" (UID: "f82dd29d-21f3-459b-a036-700eca9352f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:32.063538 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.063514    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft" (OuterVolumeSpecName: "kube-api-access-l7bft") pod "f82dd29d-21f3-459b-a036-700eca9352f2" (UID: "f82dd29d-21f3-459b-a036-700eca9352f2"). InnerVolumeSpecName "kube-api-access-l7bft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:32.063641 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.063522    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f82dd29d-21f3-459b-a036-700eca9352f2" (UID: "f82dd29d-21f3-459b-a036-700eca9352f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:32.162247 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.162206    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f82dd29d-21f3-459b-a036-700eca9352f2-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:27:32.162247 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.162235    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7bft\" (UniqueName: \"kubernetes.io/projected/f82dd29d-21f3-459b-a036-700eca9352f2-kube-api-access-l7bft\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:27:32.162247 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.162244    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f82dd29d-21f3-459b-a036-700eca9352f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:27:32.731108 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731072    2562 generic.go:356] "Generic (PLEG): container finished" podID="f82dd29d-21f3-459b-a036-700eca9352f2" containerID="b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd" exitCode=0
Sep 17 16:27:32.731490 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731124    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"
Sep 17 16:27:32.731490 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerDied","Data":"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"}
Sep 17 16:27:32.731490 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp" event={"ID":"f82dd29d-21f3-459b-a036-700eca9352f2","Type":"ContainerDied","Data":"0292d230a56707913ff0b7759e70d8e9d15aae8e49ae8e4ac916a06e3eb61a96"}
Sep 17 16:27:32.731490 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731242    2562 scope.go:117] "RemoveContainer" containerID="8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"
Sep 17 16:27:32.731698 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.731669    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:27:32.740353 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.740336    2562 scope.go:117] "RemoveContainer" containerID="b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"
Sep 17 16:27:32.747459 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.747439    2562 scope.go:117] "RemoveContainer" containerID="8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"
Sep 17 16:27:32.747730 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:27:32.747714    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58\": container with ID starting with 8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58 not found: ID does not exist" containerID="8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"
Sep 17 16:27:32.747789 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.747737    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58"} err="failed to get container status \"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58\": rpc error: code = NotFound desc = could not find container \"8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58\": container with ID starting with 8b0d933cb41811431bfa09fad009a343c6b7d2659adcf63e53daf432af6b5a58 not found: ID does not exist"
Sep 17 16:27:32.747789 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.747751    2562 scope.go:117] "RemoveContainer" containerID="b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"
Sep 17 16:27:32.747973 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:27:32.747959    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd\": container with ID starting with b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd not found: ID does not exist" containerID="b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"
Sep 17 16:27:32.748010 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.747976    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd"} err="failed to get container status \"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd\": rpc error: code = NotFound desc = could not find container \"b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd\": container with ID starting with b3e7053bee17a767596283a4ada4c8f8594863d43c0c15f57314182c76cbacbd not found: ID does not exist"
Sep 17 16:27:32.755079 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.755060    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:27:32.761968 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:32.761949    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1e5cb-predictor-79cdb767cc-t5lsp"]
Sep 17 16:27:33.478496 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:33.478462    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" path="/var/lib/kubelet/pods/f82dd29d-21f3-459b-a036-700eca9352f2/volumes"
Sep 17 16:27:37.737382 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:37.737354    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:27:37.737847 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:37.737791    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:27:47.738273 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:47.738231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:27:57.738011 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:27:57.737964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:28:07.738248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:07.738199    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 16:28:08.710703 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.710603    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:28:08.710943 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.710897    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container" containerID="cri-o://6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c" gracePeriod=30
Sep 17 16:28:08.711043 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.710945    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kube-rbac-proxy" containerID="cri-o://de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739" gracePeriod=30
Sep 17 16:28:08.897745 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.897698    2562 generic.go:356] "Generic (PLEG): container finished" podID="30952538-669f-4d74-960f-ae136009a7bc" containerID="de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739" exitCode=2
Sep 17 16:28:08.897745 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.897733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerDied","Data":"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"}
Sep 17 16:28:08.935135 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935106    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:28:08.935483 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935468    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container"
Sep 17 16:28:08.935536 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935485    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container"
Sep 17 16:28:08.935536 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935494    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kube-rbac-proxy"
Sep 17 16:28:08.935536 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935499    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kube-rbac-proxy"
Sep 17 16:28:08.935653 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935557    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kube-rbac-proxy"
Sep 17 16:28:08.935653 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.935568    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82dd29d-21f3-459b-a036-700eca9352f2" containerName="kserve-container"
Sep 17 16:28:08.938659 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.938638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:08.942212 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.942193    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-37c70-predictor-serving-cert\""
Sep 17 16:28:08.942855 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.942834    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-37c70-kube-rbac-proxy-sar-config\""
Sep 17 16:28:08.953269 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:08.953248    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:28:09.077828 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.077751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.077965 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.077835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.077965 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.077864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mc8\" (UniqueName: \"kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.178659 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.178605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mc8\" (UniqueName: \"kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.178843 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.178675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.178843 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.178731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.179398 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.179373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.181189 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.181168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.190396 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.190348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mc8\" (UniqueName: \"kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8\") pod \"error-404-isvc-37c70-predictor-5655c4d7c-tlmsr\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") " pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.248741 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.248707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.525006 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.524984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:28:09.526811 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:28:09.526783    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8efe43_779c_413c_9a5b_db90a22c4a15.slice/crio-9881190a33c24e4666ccd6541427fe740bb1d34642ee910ef146be205cc353a9 WatchSource:0}: Error finding container 9881190a33c24e4666ccd6541427fe740bb1d34642ee910ef146be205cc353a9: Status 404 returned error can't find the container with id 9881190a33c24e4666ccd6541427fe740bb1d34642ee910ef146be205cc353a9
Sep 17 16:28:09.903976 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.903894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerStarted","Data":"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"}
Sep 17 16:28:09.903976 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.903950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerStarted","Data":"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"}
Sep 17 16:28:09.904352 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.903976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerStarted","Data":"9881190a33c24e4666ccd6541427fe740bb1d34642ee910ef146be205cc353a9"}
Sep 17 16:28:09.904352 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.904044    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:09.931201 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:09.931151    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podStartSLOduration=1.93113708 podStartE2EDuration="1.93113708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:09.927855588 +0000 UTC m=+1940.783556147" watchObservedRunningTime="2026-09-17 16:28:09.93113708 +0000 UTC m=+1940.786837641"
Sep 17 16:28:10.908357 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:10.908327    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:10.909844 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:10.909815    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:11.863683 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.863654    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:28:11.899311 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.899276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config\") pod \"30952538-669f-4d74-960f-ae136009a7bc\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") "
Sep 17 16:28:11.899311 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.899308    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzgw\" (UniqueName: \"kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw\") pod \"30952538-669f-4d74-960f-ae136009a7bc\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") "
Sep 17 16:28:11.899532 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.899351    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") pod \"30952538-669f-4d74-960f-ae136009a7bc\" (UID: \"30952538-669f-4d74-960f-ae136009a7bc\") "
Sep 17 16:28:11.899668 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.899646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30952538-669f-4d74-960f-ae136009a7bc" (UID: "30952538-669f-4d74-960f-ae136009a7bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:11.901449 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.901423    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw" (OuterVolumeSpecName: "kube-api-access-xlzgw") pod "30952538-669f-4d74-960f-ae136009a7bc" (UID: "30952538-669f-4d74-960f-ae136009a7bc"). InnerVolumeSpecName "kube-api-access-xlzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:11.901449 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.901432    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30952538-669f-4d74-960f-ae136009a7bc" (UID: "30952538-669f-4d74-960f-ae136009a7bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:11.914440 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.914417    2562 generic.go:356] "Generic (PLEG): container finished" podID="30952538-669f-4d74-960f-ae136009a7bc" containerID="6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c" exitCode=0
Sep 17 16:28:11.914858 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.914468    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"
Sep 17 16:28:11.914858 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.914508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerDied","Data":"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"}
Sep 17 16:28:11.914858 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.914562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh" event={"ID":"30952538-669f-4d74-960f-ae136009a7bc","Type":"ContainerDied","Data":"85636c411c52315a5d098b02abb8c4def9d159b61bac09bf23a70ae360dd316f"}
Sep 17 16:28:11.914858 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.914588    2562 scope.go:117] "RemoveContainer" containerID="de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"
Sep 17 16:28:11.915141 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.915101    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:11.923191 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.923175    2562 scope.go:117] "RemoveContainer" containerID="6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"
Sep 17 16:28:11.930473 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.930459    2562 scope.go:117] "RemoveContainer" containerID="de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"
Sep 17 16:28:11.930753 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:11.930733    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739\": container with ID starting with de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739 not found: ID does not exist" containerID="de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"
Sep 17 16:28:11.930817 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.930760    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739"} err="failed to get container status \"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739\": rpc error: code = NotFound desc = could not find container \"de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739\": container with ID starting with de2615044381038f4c03b933383b2db7c0d88fdd50ca3a724c90073a89f9f739 not found: ID does not exist"
Sep 17 16:28:11.930817 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.930776    2562 scope.go:117] "RemoveContainer" containerID="6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"
Sep 17 16:28:11.931165 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:11.930991    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c\": container with ID starting with 6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c not found: ID does not exist" containerID="6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"
Sep 17 16:28:11.931165 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.931006    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c"} err="failed to get container status \"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c\": rpc error: code = NotFound desc = could not find container \"6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c\": container with ID starting with 6c0db2db1ef99e396f3f0a84e83455399b62947d60d09cfdea14474d8b114e7c not found: ID does not exist"
Sep 17 16:28:11.938265 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.938245    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:28:11.942883 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.942861    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26f37-predictor-64fcbb98df-qdjgh"]
Sep 17 16:28:12.000007 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.999948    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30952538-669f-4d74-960f-ae136009a7bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:12.000007 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.999968    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzgw\" (UniqueName: \"kubernetes.io/projected/30952538-669f-4d74-960f-ae136009a7bc-kube-api-access-xlzgw\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:12.000007 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:11.999979    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30952538-669f-4d74-960f-ae136009a7bc-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:13.475752 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:13.475716    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30952538-669f-4d74-960f-ae136009a7bc" path="/var/lib/kubelet/pods/30952538-669f-4d74-960f-ae136009a7bc/volumes"
Sep 17 16:28:16.919746 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:16.919710    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:16.920272 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:16.920245    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:17.738790 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:17.738758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:28:26.921160 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:26.921124    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:36.921123 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:36.921039    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:39.041850 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.041821    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:28:39.042291 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.042035    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container" containerID="cri-o://fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606" gracePeriod=30
Sep 17 16:28:39.042291 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.042113    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kube-rbac-proxy" containerID="cri-o://3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3" gracePeriod=30
Sep 17 16:28:39.273294 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273260    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:28:39.273605 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273593    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container"
Sep 17 16:28:39.273673 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273606    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container"
Sep 17 16:28:39.273673 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273628    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kube-rbac-proxy"
Sep 17 16:28:39.273673 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273634    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kube-rbac-proxy"
Sep 17 16:28:39.273795 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273696    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kube-rbac-proxy"
Sep 17 16:28:39.273795 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.273705    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30952538-669f-4d74-960f-ae136009a7bc" containerName="kserve-container"
Sep 17 16:28:39.276813 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.276791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.279556 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.279533    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6d5c6-predictor-serving-cert\""
Sep 17 16:28:39.279707 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.279555    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6d5c6-kube-rbac-proxy-sar-config\""
Sep 17 16:28:39.296706 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.292536    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:28:39.296706 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.293338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5sn5\" (UniqueName: \"kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.296706 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.293431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.296706 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.293501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.394161 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.394122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.394390 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.394205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.394390 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.394249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sn5\" (UniqueName: \"kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.394390 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:39.394271    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-serving-cert: secret "error-404-isvc-6d5c6-predictor-serving-cert" not found
Sep 17 16:28:39.394390 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:39.394361    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls podName:78ef311d-80cb-4e4c-8664-7fc6ed4f6125 nodeName:}" failed. No retries permitted until 2026-09-17 16:28:39.894339125 +0000 UTC m=+1970.750039680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls") pod "error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" (UID: "78ef311d-80cb-4e4c-8664-7fc6ed4f6125") : secret "error-404-isvc-6d5c6-predictor-serving-cert" not found
Sep 17 16:28:39.395358 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.395331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.407077 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.407056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sn5\" (UniqueName: \"kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.898814 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.898780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:39.901273 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:39.901245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") pod \"error-404-isvc-6d5c6-predictor-7555cb569-cqrlb\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") " pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:40.048520 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:40.048484    2562 generic.go:356] "Generic (PLEG): container finished" podID="78af82a4-b903-4ebd-8908-57f189e04659" containerID="3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3" exitCode=2
Sep 17 16:28:40.048904 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:40.048554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerDied","Data":"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"}
Sep 17 16:28:40.187530 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:40.187442    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:40.469229 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:40.469202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:28:40.471837 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:28:40.471801    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ef311d_80cb_4e4c_8664_7fc6ed4f6125.slice/crio-a3677c2af1f0c6690f490754a08ada74335ffff037b4dae2bf8eb23a65469a42 WatchSource:0}: Error finding container a3677c2af1f0c6690f490754a08ada74335ffff037b4dae2bf8eb23a65469a42: Status 404 returned error can't find the container with id a3677c2af1f0c6690f490754a08ada74335ffff037b4dae2bf8eb23a65469a42
Sep 17 16:28:41.054291 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.054255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerStarted","Data":"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"}
Sep 17 16:28:41.054681 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.054314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerStarted","Data":"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"}
Sep 17 16:28:41.054681 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.054332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerStarted","Data":"a3677c2af1f0c6690f490754a08ada74335ffff037b4dae2bf8eb23a65469a42"}
Sep 17 16:28:41.054681 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.054498    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:41.054681 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.054653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:41.056134 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.056110    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:28:41.087078 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:41.087027    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podStartSLOduration=2.087011978 podStartE2EDuration="2.087011978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:41.086170503 +0000 UTC m=+1971.941871088" watchObservedRunningTime="2026-09-17 16:28:41.087011978 +0000 UTC m=+1971.942712541"
Sep 17 16:28:42.058827 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:42.058779    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:28:42.732143 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:42.732090    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 16:28:42.984364 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:42.984312    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:28:43.025712 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.025689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") pod \"78af82a4-b903-4ebd-8908-57f189e04659\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") "
Sep 17 16:28:43.025845 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.025734    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config\") pod \"78af82a4-b903-4ebd-8908-57f189e04659\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") "
Sep 17 16:28:43.025845 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.025766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjt2k\" (UniqueName: \"kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k\") pod \"78af82a4-b903-4ebd-8908-57f189e04659\" (UID: \"78af82a4-b903-4ebd-8908-57f189e04659\") "
Sep 17 16:28:43.026127 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.026103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78af82a4-b903-4ebd-8908-57f189e04659" (UID: "78af82a4-b903-4ebd-8908-57f189e04659"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:43.027914 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.027893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78af82a4-b903-4ebd-8908-57f189e04659" (UID: "78af82a4-b903-4ebd-8908-57f189e04659"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:43.028008 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.027933    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k" (OuterVolumeSpecName: "kube-api-access-pjt2k") pod "78af82a4-b903-4ebd-8908-57f189e04659" (UID: "78af82a4-b903-4ebd-8908-57f189e04659"). InnerVolumeSpecName "kube-api-access-pjt2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:43.066378 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.066345    2562 generic.go:356] "Generic (PLEG): container finished" podID="78af82a4-b903-4ebd-8908-57f189e04659" containerID="fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606" exitCode=0
Sep 17 16:28:43.066708 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.066385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerDied","Data":"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"}
Sep 17 16:28:43.066708 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.066408    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"
Sep 17 16:28:43.066708 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.066430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s" event={"ID":"78af82a4-b903-4ebd-8908-57f189e04659","Type":"ContainerDied","Data":"1e995183677c42702ed65a4ce23d5ba017824c547df2f469dc5ef2529626171c"}
Sep 17 16:28:43.066708 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.066457    2562 scope.go:117] "RemoveContainer" containerID="3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"
Sep 17 16:28:43.075101 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.075078    2562 scope.go:117] "RemoveContainer" containerID="fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"
Sep 17 16:28:43.082145 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.082126    2562 scope.go:117] "RemoveContainer" containerID="3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"
Sep 17 16:28:43.082400 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:43.082379    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3\": container with ID starting with 3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3 not found: ID does not exist" containerID="3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"
Sep 17 16:28:43.082484 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.082408    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3"} err="failed to get container status \"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3\": rpc error: code = NotFound desc = could not find container \"3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3\": container with ID starting with 3f695c562be1f3afe705ee1da01f2ee3c5aefccc19eb09c45ecdc109e7bfdeb3 not found: ID does not exist"
Sep 17 16:28:43.082484 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.082431    2562 scope.go:117] "RemoveContainer" containerID="fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"
Sep 17 16:28:43.082701 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:28:43.082683    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606\": container with ID starting with fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606 not found: ID does not exist" containerID="fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"
Sep 17 16:28:43.082749 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.082708    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606"} err="failed to get container status \"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606\": rpc error: code = NotFound desc = could not find container \"fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606\": container with ID starting with fa2ac2c8b78ff31472958bbd8c00fa2760ed417c8b44de6d44c586b426d62606 not found: ID does not exist"
Sep 17 16:28:43.108664 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.108637    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:28:43.126510 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.126492    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78af82a4-b903-4ebd-8908-57f189e04659-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:43.126605 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.126513    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78af82a4-b903-4ebd-8908-57f189e04659-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:43.126605 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.126522    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjt2k\" (UniqueName: \"kubernetes.io/projected/78af82a4-b903-4ebd-8908-57f189e04659-kube-api-access-pjt2k\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:28:43.130599 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.130578    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7ce9a-predictor-5457d7f87f-c7x4s"]
Sep 17 16:28:43.474453 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:43.474419    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78af82a4-b903-4ebd-8908-57f189e04659" path="/var/lib/kubelet/pods/78af82a4-b903-4ebd-8908-57f189e04659/volumes"
Sep 17 16:28:46.920208 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:46.920172    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 16:28:47.063834 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:47.063806    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:28:47.064460 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:47.064433    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:28:56.921513 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:56.921478    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:28:57.065287 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:28:57.065251    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:29:07.065177 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:29:07.065139    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:29:17.064428 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:29:17.064388    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:29:27.064770 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:29:27.064738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:30:52.553235 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:30:52.553109    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:30:52.557239 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:30:52.557223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:35:52.582434 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:35:52.582322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:35:52.587440 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:35:52.587420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:37:54.095267 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:54.095190    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:37:54.095800 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:54.095427    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container" containerID="cri-o://edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf" gracePeriod=30
Sep 17 16:37:54.095800 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:54.095459    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kube-rbac-proxy" containerID="cri-o://020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9" gracePeriod=30
Sep 17 16:37:54.501226 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:54.501197    2562 generic.go:356] "Generic (PLEG): container finished" podID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerID="020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9" exitCode=2
Sep 17 16:37:54.501391 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:54.501244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerDied","Data":"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"}
Sep 17 16:37:57.051253 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.051225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:37:57.154387 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.154315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config\") pod \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") "
Sep 17 16:37:57.154387 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.154378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") pod \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") "
Sep 17 16:37:57.154569 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.154411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5sn5\" (UniqueName: \"kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5\") pod \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\" (UID: \"78ef311d-80cb-4e4c-8664-7fc6ed4f6125\") "
Sep 17 16:37:57.154670 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.154646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78ef311d-80cb-4e4c-8664-7fc6ed4f6125" (UID: "78ef311d-80cb-4e4c-8664-7fc6ed4f6125"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:37:57.156449 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.156424    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78ef311d-80cb-4e4c-8664-7fc6ed4f6125" (UID: "78ef311d-80cb-4e4c-8664-7fc6ed4f6125"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:37:57.156544 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.156487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5" (OuterVolumeSpecName: "kube-api-access-q5sn5") pod "78ef311d-80cb-4e4c-8664-7fc6ed4f6125" (UID: "78ef311d-80cb-4e4c-8664-7fc6ed4f6125"). InnerVolumeSpecName "kube-api-access-q5sn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:37:57.255103 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.255073    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.255103 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.255098    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.255103 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.255108    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5sn5\" (UniqueName: \"kubernetes.io/projected/78ef311d-80cb-4e4c-8664-7fc6ed4f6125-kube-api-access-q5sn5\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:37:57.516182 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.516151    2562 generic.go:356] "Generic (PLEG): container finished" podID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerID="edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf" exitCode=0
Sep 17 16:37:57.516317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.516199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerDied","Data":"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"}
Sep 17 16:37:57.516317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.516203    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"
Sep 17 16:37:57.516317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.516244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb" event={"ID":"78ef311d-80cb-4e4c-8664-7fc6ed4f6125","Type":"ContainerDied","Data":"a3677c2af1f0c6690f490754a08ada74335ffff037b4dae2bf8eb23a65469a42"}
Sep 17 16:37:57.516317 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.516274    2562 scope.go:117] "RemoveContainer" containerID="020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"
Sep 17 16:37:57.524435 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.524418    2562 scope.go:117] "RemoveContainer" containerID="edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"
Sep 17 16:37:57.531724 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.531704    2562 scope.go:117] "RemoveContainer" containerID="020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"
Sep 17 16:37:57.531945 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:37:57.531929    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9\": container with ID starting with 020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9 not found: ID does not exist" containerID="020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"
Sep 17 16:37:57.531997 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.531952    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9"} err="failed to get container status \"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9\": rpc error: code = NotFound desc = could not find container \"020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9\": container with ID starting with 020866437d3e8c92cedc4cdda5e5d191ac4f2a03634e5742e7083e43adac1dc9 not found: ID does not exist"
Sep 17 16:37:57.531997 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.531973    2562 scope.go:117] "RemoveContainer" containerID="edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"
Sep 17 16:37:57.532214 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:37:57.532196    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf\": container with ID starting with edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf not found: ID does not exist" containerID="edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"
Sep 17 16:37:57.532254 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.532221    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf"} err="failed to get container status \"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf\": rpc error: code = NotFound desc = could not find container \"edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf\": container with ID starting with edcb63ab96623d34ee54472dc76a9c419885782978c3256934895f955eaf6bcf not found: ID does not exist"
Sep 17 16:37:57.539486 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.539462    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:37:57.549809 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:57.549786    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d5c6-predictor-7555cb569-cqrlb"]
Sep 17 16:37:59.475344 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:37:59.475312    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" path="/var/lib/kubelet/pods/78ef311d-80cb-4e4c-8664-7fc6ed4f6125/volumes"
Sep 17 16:40:52.607525 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:40:52.607409    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:40:52.614122 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:40:52.614104    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:45:28.449401 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:28.449358    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:45:28.449993 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:28.449650    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container" containerID="cri-o://9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437" gracePeriod=30
Sep 17 16:45:28.449993 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:28.449739    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kube-rbac-proxy" containerID="cri-o://ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255" gracePeriod=30
Sep 17 16:45:29.446980 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:29.446947    2562 generic.go:356] "Generic (PLEG): container finished" podID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerID="ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255" exitCode=2
Sep 17 16:45:29.447150 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:29.447023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerDied","Data":"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"}
Sep 17 16:45:31.295458 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.295433    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:45:31.396983 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.396900    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls\") pod \"de8efe43-779c-413c-9a5b-db90a22c4a15\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") "
Sep 17 16:45:31.396983 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.396960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config\") pod \"de8efe43-779c-413c-9a5b-db90a22c4a15\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") "
Sep 17 16:45:31.397158 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.396999    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4mc8\" (UniqueName: \"kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8\") pod \"de8efe43-779c-413c-9a5b-db90a22c4a15\" (UID: \"de8efe43-779c-413c-9a5b-db90a22c4a15\") "
Sep 17 16:45:31.397324 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.397300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de8efe43-779c-413c-9a5b-db90a22c4a15" (UID: "de8efe43-779c-413c-9a5b-db90a22c4a15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:31.399050 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.399022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de8efe43-779c-413c-9a5b-db90a22c4a15" (UID: "de8efe43-779c-413c-9a5b-db90a22c4a15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:31.399158 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.399077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8" (OuterVolumeSpecName: "kube-api-access-g4mc8") pod "de8efe43-779c-413c-9a5b-db90a22c4a15" (UID: "de8efe43-779c-413c-9a5b-db90a22c4a15"). InnerVolumeSpecName "kube-api-access-g4mc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:31.456531 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.456496    2562 generic.go:356] "Generic (PLEG): container finished" podID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerID="9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437" exitCode=0
Sep 17 16:45:31.456699 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.456555    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"
Sep 17 16:45:31.456699 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.456591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerDied","Data":"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"}
Sep 17 16:45:31.456699 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.456681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr" event={"ID":"de8efe43-779c-413c-9a5b-db90a22c4a15","Type":"ContainerDied","Data":"9881190a33c24e4666ccd6541427fe740bb1d34642ee910ef146be205cc353a9"}
Sep 17 16:45:31.456877 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.456721    2562 scope.go:117] "RemoveContainer" containerID="ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"
Sep 17 16:45:31.465096 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.465080    2562 scope.go:117] "RemoveContainer" containerID="9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"
Sep 17 16:45:31.472966 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.472950    2562 scope.go:117] "RemoveContainer" containerID="ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"
Sep 17 16:45:31.473265 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:45:31.473243    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255\": container with ID starting with ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255 not found: ID does not exist" containerID="ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"
Sep 17 16:45:31.473354 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.473271    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255"} err="failed to get container status \"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255\": rpc error: code = NotFound desc = could not find container \"ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255\": container with ID starting with ae666e51807f56c700178543c7b1a180799bee3a30a4870da90d1bfd4a424255 not found: ID does not exist"
Sep 17 16:45:31.473354 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.473289    2562 scope.go:117] "RemoveContainer" containerID="9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"
Sep 17 16:45:31.473524 ip-10-0-143-187 kubenswrapper[2562]: E0917 16:45:31.473507    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437\": container with ID starting with 9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437 not found: ID does not exist" containerID="9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"
Sep 17 16:45:31.473566 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.473531    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437"} err="failed to get container status \"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437\": rpc error: code = NotFound desc = could not find container \"9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437\": container with ID starting with 9ccf0ec54ed91665ac7fedd6aeea0d2e034af751bbd0eb8188fd61a7c9094437 not found: ID does not exist"
Sep 17 16:45:31.480607 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.480586    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:45:31.486271 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.486249    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-37c70-predictor-5655c4d7c-tlmsr"]
Sep 17 16:45:31.498081 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.498063    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mc8\" (UniqueName: \"kubernetes.io/projected/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-api-access-g4mc8\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:45:31.498167 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.498082    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de8efe43-779c-413c-9a5b-db90a22c4a15-proxy-tls\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:45:31.498167 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:31.498092    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de8efe43-779c-413c-9a5b-db90a22c4a15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-187.ec2.internal\" DevicePath \"\""
Sep 17 16:45:33.480628 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:33.480571    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" path="/var/lib/kubelet/pods/de8efe43-779c-413c-9a5b-db90a22c4a15/volumes"
Sep 17 16:45:52.632397 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:52.632286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:45:52.639023 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:45:52.639002    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:46:03.119132 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:03.119102    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhmrn_249026f0-6f4d-43a2-96ae-bd00ac176d62/global-pull-secret-syncer/0.log"
Sep 17 16:46:03.223102 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:03.223069    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vvclt_694737e1-5cd2-4739-8254-49c167544e79/konnectivity-agent/0.log"
Sep 17 16:46:03.342331 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:03.342296    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-187.ec2.internal_0c3c972c2ec7ee80e8e1f20047dd8543/haproxy/0.log"
Sep 17 16:46:07.497453 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:07.497422    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-788bfb7cfb-d694t_30257189-69fb-4251-a895-342acd26ffd7/metrics-server/0.log"
Sep 17 16:46:07.533650 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:07.533609    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-fmssx_2d41b95c-913a-4160-bb29-4458d74a6681/monitoring-plugin/0.log"
Sep 17 16:46:07.575570 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:07.575545    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bvrr_f717488d-bc7e-4e7f-b9e0-2391f83f1584/node-exporter/0.log"
Sep 17 16:46:07.605801 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:07.605779    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bvrr_f717488d-bc7e-4e7f-b9e0-2391f83f1584/kube-rbac-proxy/0.log"
Sep 17 16:46:07.635426 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:07.635406    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bvrr_f717488d-bc7e-4e7f-b9e0-2391f83f1584/init-textfile/0.log"
Sep 17 16:46:08.487039 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.487010    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/thanos-query/0.log"
Sep 17 16:46:08.535307 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.535285    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/kube-rbac-proxy-web/0.log"
Sep 17 16:46:08.572533 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.572449    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/kube-rbac-proxy/0.log"
Sep 17 16:46:08.610781 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.610760    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/prom-label-proxy/0.log"
Sep 17 16:46:08.645177 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.645160    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/kube-rbac-proxy-rules/0.log"
Sep 17 16:46:08.685217 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:08.685192    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dfbf546f-8kvg8_1d0aedbc-b968-461c-986d-2ca53c7900a8/kube-rbac-proxy-metrics/0.log"
Sep 17 16:46:10.836928 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:10.836898    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85c8b97f4-58lxx_2edd7c14-4486-48ce-86d8-c5583cafc747/console/0.log"
Sep 17 16:46:11.506764 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.506735    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"]
Sep 17 16:46:11.507046 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507033    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507090 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507048    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507090 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507060    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container"
Sep 17 16:46:11.507090 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container"
Sep 17 16:46:11.507090 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container"
Sep 17 16:46:11.507090 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507084    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507093    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507099    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507111    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507116    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507125    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507129    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507175    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507184    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507190    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ef311d-80cb-4e4c-8664-7fc6ed4f6125" containerName="kube-rbac-proxy"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507197    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507204    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de8efe43-779c-413c-9a5b-db90a22c4a15" containerName="kserve-container"
Sep 17 16:46:11.507248 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.507209    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78af82a4-b903-4ebd-8908-57f189e04659" containerName="kube-rbac-proxy"
Sep 17 16:46:11.511215 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.511195    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.513487 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.513468    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmzkb\"/\"kube-root-ca.crt\""
Sep 17 16:46:11.514425 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.514407    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pmzkb\"/\"default-dockercfg-7gfcx\""
Sep 17 16:46:11.514533 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.514443    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pmzkb\"/\"openshift-service-ca.crt\""
Sep 17 16:46:11.523778 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.523757    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"]
Sep 17 16:46:11.611141 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.611111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-proc\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.611311 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.611148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ccj\" (UniqueName: \"kubernetes.io/projected/eb9c3a81-6622-4190-a2a5-8733ef24869c-kube-api-access-x5ccj\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.611311 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.611208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-lib-modules\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.611311 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.611287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-podres\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.611487 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.611337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-sys\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-sys\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-proc\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ccj\" (UniqueName: \"kubernetes.io/projected/eb9c3a81-6622-4190-a2a5-8733ef24869c-kube-api-access-x5ccj\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-lib-modules\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-podres\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-podres\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-sys\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.712944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-proc\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.714083 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.713076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb9c3a81-6622-4190-a2a5-8733ef24869c-lib-modules\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.723459 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.723431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ccj\" (UniqueName: \"kubernetes.io/projected/eb9c3a81-6622-4190-a2a5-8733ef24869c-kube-api-access-x5ccj\") pod \"perf-node-gather-daemonset-dqdvm\" (UID: \"eb9c3a81-6622-4190-a2a5-8733ef24869c\") " pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:11.820667 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:11.820580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:12.089745 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.089718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"]
Sep 17 16:46:12.092304 ip-10-0-143-187 kubenswrapper[2562]: W0917 16:46:12.092275    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb9c3a81_6622_4190_a2a5_8733ef24869c.slice/crio-d88d9ac0a27c76fae4f69b4df08828f16dd93a80155dc3b7e02df1bae02b7068 WatchSource:0}: Error finding container d88d9ac0a27c76fae4f69b4df08828f16dd93a80155dc3b7e02df1bae02b7068: Status 404 returned error can't find the container with id d88d9ac0a27c76fae4f69b4df08828f16dd93a80155dc3b7e02df1bae02b7068
Sep 17 16:46:12.450517 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.450494    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2qf2_d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a/dns/0.log"
Sep 17 16:46:12.473959 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.473938    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2qf2_d3695e42-8632-4bd3-b3ac-7d2c6c1fac7a/kube-rbac-proxy/0.log"
Sep 17 16:46:12.570934 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.570909    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8kxv_206bc29b-0e43-4446-9076-ad5705a7923b/dns-node-resolver/0.log"
Sep 17 16:46:12.632277 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.632246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm" event={"ID":"eb9c3a81-6622-4190-a2a5-8733ef24869c","Type":"ContainerStarted","Data":"2c18cb8dcdb58d6a2712cc6d90293d91139899b4812b1fe03a7d3d3ca5e21e65"}
Sep 17 16:46:12.632413 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.632297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm" event={"ID":"eb9c3a81-6622-4190-a2a5-8733ef24869c","Type":"ContainerStarted","Data":"d88d9ac0a27c76fae4f69b4df08828f16dd93a80155dc3b7e02df1bae02b7068"}
Sep 17 16:46:12.632413 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.632337    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:12.651885 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:12.651839    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm" podStartSLOduration=1.651825227 podStartE2EDuration="1.651825227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:46:12.650948309 +0000 UTC m=+3023.506648890" watchObservedRunningTime="2026-09-17 16:46:12.651825227 +0000 UTC m=+3023.507525788"
Sep 17 16:46:13.049593 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:13.049563    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-675ff56f96-b2755_30a8a9a2-03df-4b84-9fdf-838bd98c5339/registry/0.log"
Sep 17 16:46:13.070532 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:13.070503    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lphmx_095422d7-51d2-4c32-817f-f4ec7024face/node-ca/0.log"
Sep 17 16:46:14.350315 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:14.350286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bdqqj_557b9ce2-56a7-44c8-955c-ee140a6bcd00/serve-healthcheck-canary/0.log"
Sep 17 16:46:15.008863 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:15.008831    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2grg_9204d550-e734-47cb-9a2b-f2b789f0b26a/kube-rbac-proxy/0.log"
Sep 17 16:46:15.035018 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:15.034993    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2grg_9204d550-e734-47cb-9a2b-f2b789f0b26a/exporter/0.log"
Sep 17 16:46:15.059819 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:15.059794    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2grg_9204d550-e734-47cb-9a2b-f2b789f0b26a/extractor/0.log"
Sep 17 16:46:17.442735 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:17.442700    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6fdd8844c6-mctfb_fb6952ea-8c6c-449d-91f6-2e4b5f1c0e63/manager/0.log"
Sep 17 16:46:17.476864 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:17.476835    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-x5qcq_145e3a7e-4b06-4a71-9cd4-14cb57f2cdd2/manager/0.log"
Sep 17 16:46:17.499565 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:17.499538    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-sph6m_158921e5-4d81-4bd1-8754-bba6441f48ec/server/0.log"
Sep 17 16:46:17.588416 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:17.588384    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-7dc6c_90cc5f29-e6c9-4320-b52b-0184cbfc8605/manager/0.log"
Sep 17 16:46:18.646742 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:18.646717    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pmzkb/perf-node-gather-daemonset-dqdvm"
Sep 17 16:46:22.285169 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:22.285142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zts9m_e4e908d4-a734-42bb-aecd-1a746cdb7ee6/migrator/0.log"
Sep 17 16:46:22.322545 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:22.322515    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zts9m_e4e908d4-a734-42bb-aecd-1a746cdb7ee6/graceful-termination/0.log"
Sep 17 16:46:24.635969 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:24.635941    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:46:24.708324 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:24.708296    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/egress-router-binary-copy/0.log"
Sep 17 16:46:24.920049 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:24.919977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/cni-plugins/0.log"
Sep 17 16:46:25.053181 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.053153    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/bond-cni-plugin/0.log"
Sep 17 16:46:25.139542 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.139516    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/routeoverride-cni/0.log"
Sep 17 16:46:25.187445 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.187382    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/whereabouts-cni-bincopy/0.log"
Sep 17 16:46:25.228656 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.228632    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vbxj6_f01d3d75-6ebd-4acc-b209-50035e01489c/whereabouts-cni/0.log"
Sep 17 16:46:25.330477 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.330448    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-x9rwz_15584b8e-f9c4-4f04-89b6-7ea3e4527a7c/kube-multus/0.log"
Sep 17 16:46:25.500559 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.500529    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b9nng_f948d78e-92e1-42d9-bc1a-be5e30b8c899/network-metrics-daemon/0.log"
Sep 17 16:46:25.530871 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:25.530852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b9nng_f948d78e-92e1-42d9-bc1a-be5e30b8c899/kube-rbac-proxy/0.log"
Sep 17 16:46:27.110660 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.110629    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-controller/0.log"
Sep 17 16:46:27.153303 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.153259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/0.log"
Sep 17 16:46:27.153303 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.153279    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovn-acl-logging/1.log"
Sep 17 16:46:27.191042 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.191017    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/kube-rbac-proxy-node/0.log"
Sep 17 16:46:27.235288 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.235259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:46:27.278035 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.278010    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/northd/0.log"
Sep 17 16:46:27.318347 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.318277    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/nbdb/0.log"
Sep 17 16:46:27.377172 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.377147    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/sbdb/0.log"
Sep 17 16:46:27.454693 ip-10-0-143-187 kubenswrapper[2562]: I0917 16:46:27.454668    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qh6xq_9202fcd5-a26c-424e-a4c6-5031f21ec9bb/ovnkube-controller/0.log"