Sep 18 19:38:20.790721 ip-10-0-143-198 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:24.121200 ip-10-0-143-198 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:24.121200 ip-10-0-143-198 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:24.121200 ip-10-0-143-198 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:24.121200 ip-10-0-143-198 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:24.121200 ip-10-0-143-198 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:24.122164 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.121274    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:24.125286 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125269    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.125286 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125285    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125289    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125293    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125296    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125299    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125302    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125305    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125309    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125312    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125317    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125320    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125324    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125327    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125330    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125333    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125336    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125338    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125341    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125344    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125346    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.125350 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125349    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125352    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125354    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125357    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125360    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125374    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125377    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125380    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125382    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125385    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125388    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125390    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125393    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125395    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125398    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125401    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125404    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125406    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125409    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125411    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.125842 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125414    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125418    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125421    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125425    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125428    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125431    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125433    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125436    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125438    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125441    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125443    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125446    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125449    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125452    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125455    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125458    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125461    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125464    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125466    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.126325 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125469    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125472    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125475    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125477    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125480    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125484    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125486    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125489    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125492    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125494    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125496    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125499    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125501    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125504    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125506    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125509    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125511    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125515    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125518    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125520    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.126851 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125523    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125525    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125528    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125530    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125533    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125536    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125540    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125544    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125547    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125550    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125552    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125555    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125558    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125560    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125563    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125565    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125568    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125570    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.125572    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.127374 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126293    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126300    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126304    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126308    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126311    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126314    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126317    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126319    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126322    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126325    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126327    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126331    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126334    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126337    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126339    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126342    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126345    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126347    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126350    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126353    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.127862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126356    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126359    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126376    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126379    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126382    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126385    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126388    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126392    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126396    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126400    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126403    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126406    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126408    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126411    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126414    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126416    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126419    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126421    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126424    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126427    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.128390 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126429    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126432    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126434    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126437    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126440    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126443    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126446    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126448    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126451    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126453    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126456    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126459    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126461    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126464    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126466    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126469    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126471    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126474    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126477    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126479    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.128901 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126482    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126485    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126488    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126490    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126493    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126495    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126498    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126501    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126503    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126506    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126508    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126511    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126514    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126517    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126520    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126522    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126525    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126528    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126531    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.129423 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126533    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126536    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126538    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126544    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126547    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126550    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126553    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126555    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126558    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126560    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126563    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126565    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126568    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126571    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126573    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126576    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126580    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126584    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126587    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.129904 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126589    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126963    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126968    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126970    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126973    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126976    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126979    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126982    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126985    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126988    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126991    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126994    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.126997    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127000    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127003    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127005    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127010    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127013    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127015    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.130393 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127018    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127021    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127024    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127026    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127029    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127032    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127034    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127037    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127039    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127042    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127045    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127050    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127054    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127057    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127060    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127063    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127065    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127068    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127070    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.130905 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127073    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127077    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127080    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127086    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127089    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127092    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127095    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127098    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127101    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127104    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127108    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127111    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127114    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127117    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127120    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127122    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127125    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127128    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127130    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.131386 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127133    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127136    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127138    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127141    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127143    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127146    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127149    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127152    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127155    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127157    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127160    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127162    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127165    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127167    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127170    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127173    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127175    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127178    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127180    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127183    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.131910 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127186    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127188    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127191    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127195    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127197    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127200    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127203    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127205    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127208    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127211    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127213    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127216    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127218    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127221    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127223    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127226    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127228    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127231    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127233    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127236    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.132416 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127239    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127241    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127244    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127321    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127353    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127377    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127383    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127388    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127391    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127396    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127401    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127404    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127407    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127411    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127415    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127419    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127422    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127425    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127428    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127431    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127434    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127438    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127441    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127445    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:24.132913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127448    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127451    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127455    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127458    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127461    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127465    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127468    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127471    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127474    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127477    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127480    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127484    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127487    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127492    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127495    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127498    2567 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127501    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127506    2567 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127509    2567 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127512    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127516    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127519    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127523    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127526    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127529    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:24.133497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127532    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127535    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127538    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127542    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127545    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127548    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127551    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127554    2567 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127557    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127560    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127564    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127567    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127570    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127573    2567 flags.go:64] FLAG: --help="false"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127576    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127579    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127582    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127585    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127588    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127591    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127596    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127599    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127602    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:24.134109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127604    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127607    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127610    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127613    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127616    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127619    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127622    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127626    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127629    2567 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127632    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127635    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127638    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127643    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127646    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127649    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127651    2567 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127654    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127657    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127660    2567 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127663    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127668    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127670    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127675    2567 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127678    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127681    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:24.134738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127684    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127687    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127690    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127693    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127697    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127704    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127707    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127710    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127713    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127716    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127722    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127725    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127729    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127732    2567 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127736    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127739    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0416bf290afaea606"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127742    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127746    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127749    2567 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127752    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127755    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127758    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127761    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127764    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127768    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:24.135439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127771    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127774    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127777    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127780    2567 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127783    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127786    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127789    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127792    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127795    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127798    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127801    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127804    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127809    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127812    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127815    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127818    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127821    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127824    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127827    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127833    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127836    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127840    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127844    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127850    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127853    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:24.136167 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127856    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127859    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127862    2567 flags.go:64] FLAG: --v="2"
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127866    2567 flags.go:64] FLAG: --version="false"
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127871    2567 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127875    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.127878    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127991    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127995    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.127998    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128001    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128004    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128007    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128009    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128012    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128015    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128018    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128021    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128023    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128027    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128030    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.137042 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128033    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128036    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128039    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128041    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128044    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128048    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128052    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128057    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128060    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128062    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128067    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128071    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128074    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128076    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128080    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128082    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128085    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128088    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128090    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.137589 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128093    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128095    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128098    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128101    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128103    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128106    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128108    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128112    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128114    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128117    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128119    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128123    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128126    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128128    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128131    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128134    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128136    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128139    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128142    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128144    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.138070 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128148    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128151    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128153    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128156    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128159    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128161    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128164    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128167    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128169    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128172    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128175    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128177    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128180    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128182    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128185    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128187    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128190    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128192    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128195    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128199    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.138583 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128202    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128205    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128208    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128212    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128214    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128217    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128219    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128222    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128225    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128228    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128230    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128233    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128237    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128240    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128243    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128245    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128248    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128251    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128254    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128256    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.139100 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128259    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128262    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128264    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128267    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128269    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.128272    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.128710    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.137561    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.137579    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.137598    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.137613    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137711    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137717    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137721    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137724    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.139601 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137727    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137729    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137732    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137735    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137737    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137740    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137743    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137745    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137748    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137750    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137753    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137756    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137758    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137761    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137764    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137768    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137772    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137777    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137780    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137783    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.139980 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137786    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137789    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137792    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137795    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137797    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137801    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137805    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137809    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137812    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137815    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137817    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137820    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137823    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137826    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137828    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137831    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137834    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137836    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137840    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137844    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.140482 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137848    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137852    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137855    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137858    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137861    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137864    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137867    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137869    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137872    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137874    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137877    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137880    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137882    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137886    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137888    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137891    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137894    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137897    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137899    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137902    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.140956 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137904    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137907    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137910    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137912    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137915    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137918    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137922    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137928    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137933    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137936    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137940    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137944    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137947    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137950    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137952    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137956    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137959    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137961    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137964    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.141510 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137966    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137969    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137972    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137974    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137977    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137980    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137983    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137985    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137988    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137991    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.137995    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138000    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138004    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138008    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138011    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.141982 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138014    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.138019    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138139    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138145    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138149    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138154    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138158    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138163    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138165    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138168    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138171    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138173    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138176    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138178    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138182    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:24.142349 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138185    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138187    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138190    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138192    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138195    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138198    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138200    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138203    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138205    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138208    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138211    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138214    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138216    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138219    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138221    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138225    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138229    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138233    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138237    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138240    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:24.142741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138242    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138245    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138247    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138250    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138253    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138256    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138258    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138261    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138263    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138266    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138268    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138271    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138274    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138277    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138279    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138282    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138285    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138287    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138290    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138293    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:24.143240 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138295    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138298    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138304    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138310    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138314    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138318    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138321    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138325    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138328    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138331    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138334    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138336    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138339    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138342    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138345    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138347    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138350    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138353    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138355    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:24.143736 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138359    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138375    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138380    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138384    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138387    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138390    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138394    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138397    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138400    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138402    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138405    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138407    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138410    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138413    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138416    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138421    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138424    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138426    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138429    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138432    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:24.144216 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138434    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138437    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138439    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138442    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138446    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138450    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:24.138454    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.138461    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.138587    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.141047    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.141940    2567 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.142047    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.142087    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:24.144758 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.144064    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:24.145207 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.145189    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:24.149706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.149688    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:24.152904 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.152885    2567 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:24.153856 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.153840    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:24.156070 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.156051    2567 fs.go:135] Filesystem UUIDs: map[21eab974-6918-4e9e-b3dd-e763b11d7b8e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d1115d92-614f-4e33-ae9a-148c0f17741f:/dev/nvme0n1p3]
Sep 18 19:38:24.156138 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.156070    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:24.161231 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.161103    2567 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:24.159806928 +0000 UTC m=+1.939177501 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105870 MemoryCapacity:33164460032 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2875a80306bb98e2010465bc8615d1 SystemUUID:ec2875a8-0306-bb98-e201-0465bc8615d1 BootID:217fd9de-3714-42d1-a803-eeeafd22a89c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582230016 Type:vfs Inodes:4048396 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582230016 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:3c:c0:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:3c:c0:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:36:f9:25:a3:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164460032 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:24.161231 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.161218    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:24.161437 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.161312    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:24.162988 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.162954    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:24.163161 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.162993    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:24.164405 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.164391    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:24.164458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.164414    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:24.164458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.164426    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:24.164458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.164442    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:24.164600 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.164540    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:24.168129 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168113    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:24.168271 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168260    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:24.168343 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168306    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:24.168343 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168329    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:24.168457 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168345    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:24.168457 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168377    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:24.168457 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.168391    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:24.169347 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.169321    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:24.169436 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.169357    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:24.171976 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.171958    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:24.175020 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.175006    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:24.175087 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.175036    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:24.176202 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176191    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:24.176239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176209    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:24.176239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176218    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:24.176239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176223    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:24.176239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176232    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:24.176239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176237    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176251    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176256    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176263    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176268    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176295    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176301    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176300    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176324    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:24.176381 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.176329    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:24.178505 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.178480    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:24.178580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.178568    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:24.178705 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.178683    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:24.180386 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.180358    2567 server.go:1268] "Started kubelet"
Sep 18 19:38:24.180454 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.180425    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:24.180546 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.180498    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:24.180607 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.180567    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:24.181233 ip-10-0-143-198 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:24.181659 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.181645    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:24.182441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.182253    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:24.185014 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.184980    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:24.190453 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.190438    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:24.190529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.190487    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:24.190703 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.190683    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.190956 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.190936    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:24.190956 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.190945    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:24.191102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.190964    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:24.191102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191073    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:24.191102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191082    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:24.191491 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191466    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:24.191491 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191483    2567 factory.go:55] Registering systemd factory
Sep 18 19:38:24.191491 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191493    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:24.191707 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191688    2567 factory.go:153] Registering CRI-O factory
Sep 18 19:38:24.191744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191711    2567 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:24.191744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191726    2567 factory.go:103] Registering Raw factory
Sep 18 19:38:24.191744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.191740    2567 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:24.192169 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.192158    2567 manager.go:349] Starting recovery of all containers
Sep 18 19:38:24.192729 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.192698    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:38:24.192858 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.192836    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:38:24.216389 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.216334    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd69l"
Sep 18 19:38:24.216389 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.216379    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd69l"
Sep 18 19:38:24.223776 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.223753    2567 manager.go:354] Recovery completed
Sep 18 19:38:24.227883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.227866    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.231412 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.231394    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.231473 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.231428    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.231473 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.231439    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.232032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232014    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:24.232032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232026    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.232158 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232028    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:24.232158 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232068    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:24.232857 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232839    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.232923 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232868    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.232923 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.232880    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.233174 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.233162    2567 policy_none.go:49] "None policy: Start"
Sep 18 19:38:24.233204 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.233179    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:24.233204 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.233190    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:24.234541 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.234529    2567 policy_none.go:47] "Start"
Sep 18 19:38:24.282111 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282093    2567 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.282141    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282154    2567 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282593    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282615    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282713    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282806    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.282817    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.283430    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:24.285090 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.283483    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.345947 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.345904    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:24.347200 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.347179    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:24.347301 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.347238    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:24.347340 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.347305    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:24.347391 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.347345    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:24.349329 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.349302    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:24.383334 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.383207    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.384302 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.384287    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.384382 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.384316    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.384382 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.384326    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.384462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.384399    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.392788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.392768    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.392843 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.392794    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-198.ec2.internal\": node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.407324 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.407294    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.447778 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.447727    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"]
Sep 18 19:38:24.447866 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.447833    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.448896 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.448882    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.448965 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.448912    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.448965 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.448921    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.450191 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.450179    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.450316 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.450292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.450353 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.450324    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.450994 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.450981    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.451050 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.451003    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.451050 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.451013    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.451122 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.450984    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.451122 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.451076    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.451122 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.451086    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.452226 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.452206    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.452283 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.452238    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:24.452966 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.452950    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:24.453049 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.452974    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:24.453049 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.452984    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:24.470014 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.469994    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-198.ec2.internal\" not found" node="ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.474001 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.473982    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-198.ec2.internal\" not found" node="ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.492340 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.492315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.492433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.492346    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ff6efb868a47e5add55da0b50fc28c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-198.ec2.internal\" (UID: \"49ff6efb868a47e5add55da0b50fc28c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.492433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.492375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.507641 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.507622    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.593482 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ff6efb868a47e5add55da0b50fc28c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-198.ec2.internal\" (UID: \"49ff6efb868a47e5add55da0b50fc28c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.593573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.593573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.593573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.593573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593551    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/49ff6efb868a47e5add55da0b50fc28c-config\") pod \"kube-apiserver-proxy-ip-10-0-143-198.ec2.internal\" (UID: \"49ff6efb868a47e5add55da0b50fc28c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.593705 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.593570    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21329fb634b2607fe3f26bd3c22bbfe7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal\" (UID: \"21329fb634b2607fe3f26bd3c22bbfe7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.608607 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.608574    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.709435 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.709354    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.774575 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.774548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.778344 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:24.778318    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:24.810473 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.810433    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:24.910954 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:24.910910    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.011481 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.011402    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.111904 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.111858    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.142048 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.142015    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:25.142520 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.142181    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:25.212357 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.212314    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.223495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.223464    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:24 +0000 UTC" deadline="2028-05-28 08:51:15.698972052 +0000 UTC"
Sep 18 19:38:25.223495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.223491    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14821h12m50.475483938s"
Sep 18 19:38:25.313296 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.313262    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.376934 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.376905    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:25.413923 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.413894    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.440460 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:25.440423    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ff6efb868a47e5add55da0b50fc28c.slice/crio-cd81e662083f2256832be2deb5a3eafcf8661990e7d56f4a0ed37516ef27c677 WatchSource:0}: Error finding container cd81e662083f2256832be2deb5a3eafcf8661990e7d56f4a0ed37516ef27c677: Status 404 returned error can't find the container with id cd81e662083f2256832be2deb5a3eafcf8661990e7d56f4a0ed37516ef27c677
Sep 18 19:38:25.440886 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:25.440867    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21329fb634b2607fe3f26bd3c22bbfe7.slice/crio-f1686dd31fdc6e89cd15d495fa8d14b47ef035c53e1a8d4be723f4b7bdf37ed8 WatchSource:0}: Error finding container f1686dd31fdc6e89cd15d495fa8d14b47ef035c53e1a8d4be723f4b7bdf37ed8: Status 404 returned error can't find the container with id f1686dd31fdc6e89cd15d495fa8d14b47ef035c53e1a8d4be723f4b7bdf37ed8
Sep 18 19:38:25.446168 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.446149    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:25.514454 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.514413    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.567508 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.567431    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:25.615310 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:25.615282    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-198.ec2.internal\" not found"
Sep 18 19:38:25.631049 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.631028    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:25.691091 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.691061    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:25.697583 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.697560    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:25.698787 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.698772    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal"
Sep 18 19:38:25.708945 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:25.708918    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:26.058998 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.058916    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:26.169808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.169778    2567 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:26.174495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.174466    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:26.174954 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.174909    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff","openshift-cluster-node-tuning-operator/tuned-dcpds","openshift-dns/node-resolver-768dq","openshift-image-registry/node-ca-b6qv7","openshift-multus/multus-6vr48","openshift-network-operator/iptables-alerter-5gtzj","kube-system/konnectivity-agent-csv52","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal","openshift-multus/multus-additional-cni-plugins-27sfw","openshift-multus/network-metrics-daemon-m6p9z","openshift-network-diagnostics/network-check-target-5qnvn","openshift-ovn-kubernetes/ovnkube-node-b7nd9","kube-system/global-pull-secret-syncer-s98hl"]
Sep 18 19:38:26.178445 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.177778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.181675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.179902    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:26.181675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.180216    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.181675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.180463    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.181675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.180683    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59szm\""
Sep 18 19:38:26.181675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.181066    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.183443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.182319    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.183443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.183060    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.183443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.183127    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.183443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.183317    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:26.183443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.183358    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:26.183813 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.183536    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:26.184188 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.184116    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q9qnn\""
Sep 18 19:38:26.184268 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.184124    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:26.184788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.184765    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.187962 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.185333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.187962 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.185539    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvxzv\""
Sep 18 19:38:26.187962 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.185670    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.187962 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.187251    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.189579    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.189795    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.189822    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sndxv\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190034    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190061    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190306    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tgsvg\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190423    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.190581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.190576    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:26.192957 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.192936    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ttxcp\""
Sep 18 19:38:26.193069 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.192943    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.193131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.193104    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.193246 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.193108    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:26.193400 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.193383    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.193505 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.193481    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:26.193769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.193753    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:26.196026 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.196007    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.199425 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.198888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.201505 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.201403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.203308 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.203287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-etc-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.203479 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.203462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcea9b5-b950-4d8b-a1e5-03c410160484-host-slash\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.203564 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.203535    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.203800 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.203771    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:26.203942 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.203579    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-sys\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.204077 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-netns\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.204229 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-env-overrides\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.204324 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/991a6b3a-db32-4391-ab67-00f063ea5999-ovn-node-metrics-cert\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.204432 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-var-lib-kubelet\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.204675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204660    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-etc-kubernetes\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.204772 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-conf\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.204867 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.204855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-var-lib-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.205210 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70ded459-00cc-4455-a8a4-a080de66c481-hosts-file\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.205320 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-multus-certs\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.205438 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.205535 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-system-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.205637 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-systemd-units\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.205700 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-bin\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.205752 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a369c464-bec9-42e5-87ef-63208b3d90f6-serviceca\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.205803 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-etc-tuned\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.205803 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-kubelet\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2zz\" (UniqueName: \"kubernetes.io/projected/a369c464-bec9-42e5-87ef-63208b3d90f6-kube-api-access-bz2zz\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-systemd\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205873    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-script-lib\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-cni-binary-copy\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-daemon-config\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.205974    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:26.206003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.205988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-slash\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-netd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a369c464-bec9-42e5-87ef-63208b3d90f6-host\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-bin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysconfig\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-run\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206159    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-lib-modules\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-node-log\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206261    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqx6f\" (UniqueName: \"kubernetes.io/projected/70ded459-00cc-4455-a8a4-a080de66c481-kube-api-access-qqx6f\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-cnibin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206331    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-socket-dir-parent\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-kubelet\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcea9b5-b950-4d8b-a1e5-03c410160484-iptables-alerter-script\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.206441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-systemd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqkn\" (UniqueName: \"kubernetes.io/projected/06f1c675-1a73-4385-864a-65d59dde6ad4-kube-api-access-tcqkn\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-dbus\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvw7\" (UniqueName: \"kubernetes.io/projected/9434d76d-f399-40a7-912b-bdd6e219800e-kube-api-access-zdvw7\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206544    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4gv\" (UniqueName: \"kubernetes.io/projected/991a6b3a-db32-4391-ab67-00f063ea5999-kube-api-access-7h4gv\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206579    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206602    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-netns\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljv2w\" (UniqueName: \"kubernetes.io/projected/adcea9b5-b950-4d8b-a1e5-03c410160484-kube-api-access-ljv2w\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-log-socket\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ded459-00cc-4455-a8a4-a080de66c481-tmp-dir\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-k8s-cni-cncf-io\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206905    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-modprobe-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206928    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-kubernetes\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-host\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207088 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206977    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-tmp\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.206995    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207178    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207202    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xr62l\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207298    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207444    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-ovn\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207500    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-config\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207553    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-multus\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-hostroot\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207602    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-kubelet-config\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-conf-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207706    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207717    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dv5gn\""
Sep 18 19:38:26.207959 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.207809    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7gsv\""
Sep 18 19:38:26.215671 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.215493    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:26.228879 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.228857    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgtxx"
Sep 18 19:38:26.240490 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.239972    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgtxx"
Sep 18 19:38:26.292261 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.292175    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-conf\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-var-lib-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70ded459-00cc-4455-a8a4-a080de66c481-hosts-file\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-multus-certs\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309815    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b69094ee-26c9-4c0b-a6c5-042b65834156-konnectivity-ca\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-system-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309951    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-systemd-units\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.309981    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-bin\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a369c464-bec9-42e5-87ef-63208b3d90f6-serviceca\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310037    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-registration-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-etc-tuned\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-kubelet\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2zz\" (UniqueName: \"kubernetes.io/projected/a369c464-bec9-42e5-87ef-63208b3d90f6-kube-api-access-bz2zz\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70ded459-00cc-4455-a8a4-a080de66c481-hosts-file\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.311993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-multus-certs\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-bin\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-systemd\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-script-lib\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-conf\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-cni-binary-copy\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-systemd\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-daemon-config\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b69094ee-26c9-4c0b-a6c5-042b65834156-agent-certs\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310611    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-slash\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310618    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-netd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310672    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a369c464-bec9-42e5-87ef-63208b3d90f6-host\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcgk\" (UniqueName: \"kubernetes.io/projected/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-kube-api-access-ppcgk\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-bin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310791    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysconfig\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.312883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-var-lib-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-run\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310841    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-lib-modules\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-slash\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-node-log\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310957    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310980    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-lib-modules\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.310990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqx6f\" (UniqueName: \"kubernetes.io/projected/70ded459-00cc-4455-a8a4-a080de66c481-kube-api-access-qqx6f\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-cnibin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-socket-dir-parent\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysconfig\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-kubelet\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311109    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-run\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcea9b5-b950-4d8b-a1e5-03c410160484-iptables-alerter-script\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-systemd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311161    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-cni-binary-copy\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqkn\" (UniqueName: \"kubernetes.io/projected/06f1c675-1a73-4385-864a-65d59dde6ad4-kube-api-access-tcqkn\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-dbus\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.313687 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-daemon-config\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-cni-netd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-cnibin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-systemd\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311360    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cnibin\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvw7\" (UniqueName: \"kubernetes.io/projected/9434d76d-f399-40a7-912b-bdd6e219800e-kube-api-access-zdvw7\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4gv\" (UniqueName: \"kubernetes.io/projected/991a6b3a-db32-4391-ab67-00f063ea5999-kube-api-access-7h4gv\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311459    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a369c464-bec9-42e5-87ef-63208b3d90f6-host\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311486    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-script-lib\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311499    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-node-log\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-netns\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-dbus\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311640    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcea9b5-b950-4d8b-a1e5-03c410160484-iptables-alerter-script\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.314359 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.311651    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.311729    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:26.811708396 +0000 UTC m=+4.591078979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-kubelet\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311825    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-netns\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-bin\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-socket-dir-parent\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.311992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-system-cni-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv2w\" (UniqueName: \"kubernetes.io/projected/adcea9b5-b950-4d8b-a1e5-03c410160484-kube-api-access-ljv2w\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-log-socket\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-sysctl-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ded459-00cc-4455-a8a4-a080de66c481-tmp-dir\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-k8s-cni-cncf-io\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312259    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-log-socket\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-sys-fs\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-run-k8s-cni-cncf-io\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglkj\" (UniqueName: \"kubernetes.io/projected/b8030510-12d0-4d39-af1f-978983b22e86-kube-api-access-hglkj\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-systemd-units\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312682    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-kubelet\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnk22\" (UniqueName: \"kubernetes.io/projected/695e01df-28bb-4aed-9e24-d53720c990fc-kube-api-access-cnk22\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-modprobe-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a369c464-bec9-42e5-87ef-63208b3d90f6-serviceca\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-kubernetes\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-host\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312950    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-kubernetes\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.315876 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.312956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-tmp\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-ovn\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313051    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-config\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313073    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-etc-modprobe-d\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-multus\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-host\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-hostroot\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313134    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-run-ovn\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313131    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313151    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-host-var-lib-cni-multus\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-hostroot\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-kubelet-config\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ded459-00cc-4455-a8a4-a080de66c481-tmp-dir\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-conf-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-kubelet-config\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-socket-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313538    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-multus-conf-dir\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-device-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.316739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-ovnkube-config\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-etc-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcea9b5-b950-4d8b-a1e5-03c410160484-host-slash\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313790    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-etc-openvswitch\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-sys\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313833    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcea9b5-b950-4d8b-a1e5-03c410160484-host-slash\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313881    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-netns\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313906    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-env-overrides\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/991a6b3a-db32-4391-ab67-00f063ea5999-ovn-node-metrics-cert\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.313951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/991a6b3a-db32-4391-ab67-00f063ea5999-host-run-netns\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-var-lib-kubelet\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-etc-kubernetes\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-sys\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06f1c675-1a73-4385-864a-65d59dde6ad4-etc-kubernetes\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9434d76d-f399-40a7-912b-bdd6e219800e-var-lib-kubelet\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.314357    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/991a6b3a-db32-4391-ab67-00f063ea5999-env-overrides\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.320462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-etc-tuned\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.325517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.320612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9434d76d-f399-40a7-912b-bdd6e219800e-tmp\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.326456 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.321794    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvw7\" (UniqueName: \"kubernetes.io/projected/9434d76d-f399-40a7-912b-bdd6e219800e-kube-api-access-zdvw7\") pod \"tuned-dcpds\" (UID: \"9434d76d-f399-40a7-912b-bdd6e219800e\") " pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.326456 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.321857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2zz\" (UniqueName: \"kubernetes.io/projected/a369c464-bec9-42e5-87ef-63208b3d90f6-kube-api-access-bz2zz\") pod \"node-ca-b6qv7\" (UID: \"a369c464-bec9-42e5-87ef-63208b3d90f6\") " pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.326456 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.324291    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4gv\" (UniqueName: \"kubernetes.io/projected/991a6b3a-db32-4391-ab67-00f063ea5999-kube-api-access-7h4gv\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.326456 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.325687    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqkn\" (UniqueName: \"kubernetes.io/projected/06f1c675-1a73-4385-864a-65d59dde6ad4-kube-api-access-tcqkn\") pod \"multus-6vr48\" (UID: \"06f1c675-1a73-4385-864a-65d59dde6ad4\") " pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.327004 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.326781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/991a6b3a-db32-4391-ab67-00f063ea5999-ovn-node-metrics-cert\") pod \"ovnkube-node-b7nd9\" (UID: \"991a6b3a-db32-4391-ab67-00f063ea5999\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.328929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.328703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv2w\" (UniqueName: \"kubernetes.io/projected/adcea9b5-b950-4d8b-a1e5-03c410160484-kube-api-access-ljv2w\") pod \"iptables-alerter-5gtzj\" (UID: \"adcea9b5-b950-4d8b-a1e5-03c410160484\") " pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.330260 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.329806    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqx6f\" (UniqueName: \"kubernetes.io/projected/70ded459-00cc-4455-a8a4-a080de66c481-kube-api-access-qqx6f\") pod \"node-resolver-768dq\" (UID: \"70ded459-00cc-4455-a8a4-a080de66c481\") " pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.352954 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.352903    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal" event={"ID":"49ff6efb868a47e5add55da0b50fc28c","Type":"ContainerStarted","Data":"cd81e662083f2256832be2deb5a3eafcf8661990e7d56f4a0ed37516ef27c677"}
Sep 18 19:38:26.354744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.354638    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal" event={"ID":"21329fb634b2607fe3f26bd3c22bbfe7","Type":"ContainerStarted","Data":"f1686dd31fdc6e89cd15d495fa8d14b47ef035c53e1a8d4be723f4b7bdf37ed8"}
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cnibin\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414664    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414692    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-sys-fs\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglkj\" (UniqueName: \"kubernetes.io/projected/b8030510-12d0-4d39-af1f-978983b22e86-kube-api-access-hglkj\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnk22\" (UniqueName: \"kubernetes.io/projected/695e01df-28bb-4aed-9e24-d53720c990fc-kube-api-access-cnk22\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-socket-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414821    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-device-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b69094ee-26c9-4c0b-a6c5-042b65834156-konnectivity-ca\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.417733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414953    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-registration-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.414992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b69094ee-26c9-4c0b-a6c5-042b65834156-agent-certs\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.415019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.415045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcgk\" (UniqueName: \"kubernetes.io/projected/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-kube-api-access-ppcgk\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.415549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-etc-selinux\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.415594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cnibin\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.415813    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.415913    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.415969    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:26.915948633 +0000 UTC m=+4.695319194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.416128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.416661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.416722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-sys-fs\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.416903    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.417118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-socket-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.417183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-device-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.417463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8030510-12d0-4d39-af1f-978983b22e86-registration-dir\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.418559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.417586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.419313 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.417922    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.419313 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.418066    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b69094ee-26c9-4c0b-a6c5-042b65834156-konnectivity-ca\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.423164 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.422576    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:26.423164 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.422597    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:26.423164 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.422611    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:26.423164 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.422616    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b69094ee-26c9-4c0b-a6c5-042b65834156-agent-certs\") pod \"konnectivity-agent-csv52\" (UID: \"b69094ee-26c9-4c0b-a6c5-042b65834156\") " pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.423164 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.422668    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:26.922653062 +0000 UTC m=+4.702023627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:26.425435 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.425388    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcgk\" (UniqueName: \"kubernetes.io/projected/1608a4d9-2d53-4b44-b6f1-b7159dce3a8f-kube-api-access-ppcgk\") pod \"multus-additional-cni-plugins-27sfw\" (UID: \"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f\") " pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.426566 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.426546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnk22\" (UniqueName: \"kubernetes.io/projected/695e01df-28bb-4aed-9e24-d53720c990fc-kube-api-access-cnk22\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.426759 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.426737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglkj\" (UniqueName: \"kubernetes.io/projected/b8030510-12d0-4d39-af1f-978983b22e86-kube-api-access-hglkj\") pod \"aws-ebs-csi-driver-node-bgrff\" (UID: \"b8030510-12d0-4d39-af1f-978983b22e86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.493913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.493880    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5gtzj"
Sep 18 19:38:26.503143 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.503115    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:26.516016 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.515989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dcpds"
Sep 18 19:38:26.519751 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.519710    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-768dq"
Sep 18 19:38:26.527829 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.527807    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6qv7"
Sep 18 19:38:26.535886 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.535865    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vr48"
Sep 18 19:38:26.545532 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.545507    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:26.552253 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.552229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff"
Sep 18 19:38:26.566936 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.566863    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27sfw"
Sep 18 19:38:26.818498 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.818424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:26.818625 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.818575    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:26.818672 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.818642    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:27.818624841 +0000 UTC m=+5.597995402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:26.918837 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:26.918801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:26.919022 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.918928    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:26.919022 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:26.918986    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:27.918969898 +0000 UTC m=+5.698340465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:27.019729 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.019689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:27.019898 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.019862    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:27.019898 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.019884    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:27.019898 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.019897    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:27.020037 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.019964    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:28.019946947 +0000 UTC m=+5.799317511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:27.243656 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.243567    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:26 +0000 UTC" deadline="2028-06-01 14:59:23.114113884 +0000 UTC"
Sep 18 19:38:27.243656 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.243601    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14923h20m55.870517086s"
Sep 18 19:38:27.347614 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.347583    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:27.347765 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.347685    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:27.491069 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.491041    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69094ee_26c9_4c0b_a6c5_042b65834156.slice/crio-92a1a59c92ac95dac9d35f0907975f263c2371c3d22c57c3e68ab8aab6d18025 WatchSource:0}: Error finding container 92a1a59c92ac95dac9d35f0907975f263c2371c3d22c57c3e68ab8aab6d18025: Status 404 returned error can't find the container with id 92a1a59c92ac95dac9d35f0907975f263c2371c3d22c57c3e68ab8aab6d18025
Sep 18 19:38:27.492425 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.492398    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1608a4d9_2d53_4b44_b6f1_b7159dce3a8f.slice/crio-a112a7ad730cc788c45d80630d034aa1355aa48ca16559000005a9234215a0ea WatchSource:0}: Error finding container a112a7ad730cc788c45d80630d034aa1355aa48ca16559000005a9234215a0ea: Status 404 returned error can't find the container with id a112a7ad730cc788c45d80630d034aa1355aa48ca16559000005a9234215a0ea
Sep 18 19:38:27.494181 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.494161    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ded459_00cc_4455_a8a4_a080de66c481.slice/crio-7d022b4990db0c069f59409a9ac3595be527a410134f9999666dd3457eb4edf1 WatchSource:0}: Error finding container 7d022b4990db0c069f59409a9ac3595be527a410134f9999666dd3457eb4edf1: Status 404 returned error can't find the container with id 7d022b4990db0c069f59409a9ac3595be527a410134f9999666dd3457eb4edf1
Sep 18 19:38:27.515823 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.515798    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f1c675_1a73_4385_864a_65d59dde6ad4.slice/crio-fdf65140f0687ebb55c73a8c4a2cb2f02721987d53f83410ff8261471f68f292 WatchSource:0}: Error finding container fdf65140f0687ebb55c73a8c4a2cb2f02721987d53f83410ff8261471f68f292: Status 404 returned error can't find the container with id fdf65140f0687ebb55c73a8c4a2cb2f02721987d53f83410ff8261471f68f292
Sep 18 19:38:27.516744 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.516726    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8030510_12d0_4d39_af1f_978983b22e86.slice/crio-3aaee6d383b97840a162498e47285ebaaf36d8b4cb9bc411e2f9708a90de13b6 WatchSource:0}: Error finding container 3aaee6d383b97840a162498e47285ebaaf36d8b4cb9bc411e2f9708a90de13b6: Status 404 returned error can't find the container with id 3aaee6d383b97840a162498e47285ebaaf36d8b4cb9bc411e2f9708a90de13b6
Sep 18 19:38:27.522741 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.522716    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991a6b3a_db32_4391_ab67_00f063ea5999.slice/crio-04586ca90410f39e2e8953fbbe7bde6abdd53d6e38f20754a134006a22337387 WatchSource:0}: Error finding container 04586ca90410f39e2e8953fbbe7bde6abdd53d6e38f20754a134006a22337387: Status 404 returned error can't find the container with id 04586ca90410f39e2e8953fbbe7bde6abdd53d6e38f20754a134006a22337387
Sep 18 19:38:27.524965 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.524437    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda369c464_bec9_42e5_87ef_63208b3d90f6.slice/crio-f838c42626b6f8e244c3ed11e4524f196efcd853a799837638ba44582273403a WatchSource:0}: Error finding container f838c42626b6f8e244c3ed11e4524f196efcd853a799837638ba44582273403a: Status 404 returned error can't find the container with id f838c42626b6f8e244c3ed11e4524f196efcd853a799837638ba44582273403a
Sep 18 19:38:27.525213 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:27.525191    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcea9b5_b950_4d8b_a1e5_03c410160484.slice/crio-3946ac0e2acabef36f07e2a043858c9291ba0d70e4af94457f65cdde6c0e7914 WatchSource:0}: Error finding container 3946ac0e2acabef36f07e2a043858c9291ba0d70e4af94457f65cdde6c0e7914: Status 404 returned error can't find the container with id 3946ac0e2acabef36f07e2a043858c9291ba0d70e4af94457f65cdde6c0e7914
Sep 18 19:38:27.825074 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.825048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:27.825176 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.825166    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:27.825230 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.825216    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:29.825204394 +0000 UTC m=+7.604574954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:27.926171 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:27.926135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:27.926337 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.926263    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:27.926420 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:27.926337    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:29.92631428 +0000 UTC m=+7.705684863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:28.027252 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.027060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:28.027434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.027271    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:28.027434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.027300    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:28.027434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.027313    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:28.027434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.027400    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.027379231 +0000 UTC m=+7.806749806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:28.244968 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.244884    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:26 +0000 UTC" deadline="2028-05-20 17:58:05.070105877 +0000 UTC"
Sep 18 19:38:28.244968 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.244966    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h19m36.82514409s"
Sep 18 19:38:28.349023 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.348383    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:28.349023 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.348404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:28.349023 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.348578    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:28.349023 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:28.348693    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:28.378360 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.378330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerStarted","Data":"a112a7ad730cc788c45d80630d034aa1355aa48ca16559000005a9234215a0ea"}
Sep 18 19:38:28.385402 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.385284    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csv52" event={"ID":"b69094ee-26c9-4c0b-a6c5-042b65834156","Type":"ContainerStarted","Data":"92a1a59c92ac95dac9d35f0907975f263c2371c3d22c57c3e68ab8aab6d18025"}
Sep 18 19:38:28.401736 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.400903    2567 generic.go:356] "Generic (PLEG): container finished" podID="21329fb634b2607fe3f26bd3c22bbfe7" containerID="0737ea2f893f29c59c3d5b92371f8fa6c73a0c213cc3c985c5f914ca8903761f" exitCode=0
Sep 18 19:38:28.401736 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.400986    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal" event={"ID":"21329fb634b2607fe3f26bd3c22bbfe7","Type":"ContainerDied","Data":"0737ea2f893f29c59c3d5b92371f8fa6c73a0c213cc3c985c5f914ca8903761f"}
Sep 18 19:38:28.419536 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.419509    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5gtzj" event={"ID":"adcea9b5-b950-4d8b-a1e5-03c410160484","Type":"ContainerStarted","Data":"3946ac0e2acabef36f07e2a043858c9291ba0d70e4af94457f65cdde6c0e7914"}
Sep 18 19:38:28.431859 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.429028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6qv7" event={"ID":"a369c464-bec9-42e5-87ef-63208b3d90f6","Type":"ContainerStarted","Data":"f838c42626b6f8e244c3ed11e4524f196efcd853a799837638ba44582273403a"}
Sep 18 19:38:28.431859 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.431127    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vr48" event={"ID":"06f1c675-1a73-4385-864a-65d59dde6ad4","Type":"ContainerStarted","Data":"fdf65140f0687ebb55c73a8c4a2cb2f02721987d53f83410ff8261471f68f292"}
Sep 18 19:38:28.446244 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.445958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-768dq" event={"ID":"70ded459-00cc-4455-a8a4-a080de66c481","Type":"ContainerStarted","Data":"7d022b4990db0c069f59409a9ac3595be527a410134f9999666dd3457eb4edf1"}
Sep 18 19:38:28.490313 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.490283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal" event={"ID":"49ff6efb868a47e5add55da0b50fc28c","Type":"ContainerStarted","Data":"44cd9c7a2a53e8fe1883faba37c0980726e58fb59bb414086af56b21c2be0a3a"}
Sep 18 19:38:28.507868 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.507800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcpds" event={"ID":"9434d76d-f399-40a7-912b-bdd6e219800e","Type":"ContainerStarted","Data":"3facb94069869751f8fd801f7f8f385b01b1e0df6affb0262b29ec23e9f895fa"}
Sep 18 19:38:28.529756 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.529352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"04586ca90410f39e2e8953fbbe7bde6abdd53d6e38f20754a134006a22337387"}
Sep 18 19:38:28.539499 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:28.539408    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff" event={"ID":"b8030510-12d0-4d39-af1f-978983b22e86","Type":"ContainerStarted","Data":"3aaee6d383b97840a162498e47285ebaaf36d8b4cb9bc411e2f9708a90de13b6"}
Sep 18 19:38:29.348217 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:29.348182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:29.348827 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:29.348323    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:29.566883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:29.566800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal" event={"ID":"21329fb634b2607fe3f26bd3c22bbfe7","Type":"ContainerStarted","Data":"966df0761b3c6ba9a38ee57b6e895c041bf7384a0be7abda2b9fd5da99a0dc90"}
Sep 18 19:38:29.582824 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:29.582756    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-198.ec2.internal" podStartSLOduration=4.582733526 podStartE2EDuration="4.582733526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:28.503684527 +0000 UTC m=+6.283055113" watchObservedRunningTime="2026-09-18 19:38:29.582733526 +0000 UTC m=+7.362104110"
Sep 18 19:38:29.841842 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:29.841724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:29.842025 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:29.841958    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:29.842025 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:29.842022    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.842004319 +0000 UTC m=+11.621374886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:29.942944 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:29.942902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:29.943105 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:29.943065    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:29.943188 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:29.943131    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.943111181 +0000 UTC m=+11.722481757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.044038 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:30.043996    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:30.044243 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.044227    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:30.044299 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.044253    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:30.044299 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.044265    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.044424 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.044323    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:34.044305843 +0000 UTC m=+11.823676425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.349073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:30.348472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:30.349073 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.348617    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:30.349073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:30.348941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:30.349073 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:30.349031    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:31.347886 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:31.347851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:31.348059 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:31.347993    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:32.349733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:32.349025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:32.349733 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:32.349179    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:32.349733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:32.349582    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:32.349733 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:32.349688    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:33.348187 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:33.347924    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:33.348187 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:33.348059    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:33.880158 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:33.879991    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:33.880158 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:33.880062    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.880158 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:33.880133    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:41.880114006 +0000 UTC m=+19.659484584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.980983 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:33.980386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:33.980983 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:33.980576    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.980983 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:33.980651    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:41.980629857 +0000 UTC m=+19.760000420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:34.081495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:34.081450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:34.081694 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.081682    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:34.081754 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.081704    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:34.081754 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.081718    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:34.081859 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.081789    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:42.081768957 +0000 UTC m=+19.861139521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:34.350157 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:34.349493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:34.350157 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.349605    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:34.350157 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:34.349971    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:34.350157 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:34.350069    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:35.348862 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:35.348677    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:35.349304 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:35.348867    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:36.348838 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:36.348312    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:36.348838 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:36.348486    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:36.348838 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:36.348536    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:36.348838 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:36.348663    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:37.348070 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:37.348033    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:37.348227 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:37.348159    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:38.347845 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:38.347813    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:38.348279 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:38.347813    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:38.348279 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:38.347950    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:38.348279 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:38.348026    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:39.348320 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:39.348246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:39.348758 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:39.348380    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:40.348163 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:40.348127    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:40.348340 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:40.348130    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:40.348703 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:40.348343    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:40.348758 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:40.348741    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:41.347523 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:41.347483    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:41.347717 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:41.347617    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:41.938272 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:41.938063    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:41.938722 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:41.938225    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:41.938722 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:41.938390    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.938355545 +0000 UTC m=+35.717726108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:42.039574 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:42.039535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:42.039757 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.039713    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:42.039819 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.039794    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.039772436 +0000 UTC m=+35.819143022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:42.140041 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:42.140003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:42.140245 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.140219    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:42.140308 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.140252    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:42.140308 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.140266    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:42.140415 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.140335    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.140315202 +0000 UTC m=+35.919685779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:42.347645 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:42.347613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:42.347645 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:42.347642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:42.347901 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.347741    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:42.347901 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:42.347831    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:43.348224 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:43.348161    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:43.348689 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:43.348304    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:44.348994 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:44.348958    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:44.349487 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:44.349056    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:44.349487 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:44.349111    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:44.349487 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:44.349195    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:45.347595 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.347574    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:45.348292 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:45.347874    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:45.600410 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.600339    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dcpds" event={"ID":"9434d76d-f399-40a7-912b-bdd6e219800e","Type":"ContainerStarted","Data":"14ababac56e11bc36dcdfb342b15c91c1f6534c6b521d5c5b19d4139cb35de97"}
Sep 18 19:38:45.603411 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.603390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"a702d86fe8c94be834762f658eac31d41cc5d9cc0b3c54852effba5bb3b02b4c"}
Sep 18 19:38:45.603499 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.603466    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"3651225efe6fcff7dddd16830fbb38b8c6f042c9f1033c39c140f5cc6afa9da0"}
Sep 18 19:38:45.603565 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.603513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"0b4d53df1d3647197e7c557db9f4a1b1f76b81424bc5f47165eefee8f29cc8c7"}
Sep 18 19:38:45.603613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.603569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"634ebb9b5483542ac57033156af863b31be730b353456c81305c0fe83427f75b"}
Sep 18 19:38:45.603613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.603607    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"ef00b32e7de16015ddca60e9afde2b17d4cbad8369dec381ddc1b5ed42d43b82"}
Sep 18 19:38:45.604896 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.604870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff" event={"ID":"b8030510-12d0-4d39-af1f-978983b22e86","Type":"ContainerStarted","Data":"0ce54c5d3ac953921a65efbb8f8aa96dda1b4ab3f0b9499e0b11931318ef3556"}
Sep 18 19:38:45.606470 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.606445    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="8458adbe9dc39de139e40b81fc563b5d39452aa060eedae263922ee753ec6327" exitCode=0
Sep 18 19:38:45.606566 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.606510    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"8458adbe9dc39de139e40b81fc563b5d39452aa060eedae263922ee753ec6327"}
Sep 18 19:38:45.609704 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.609674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csv52" event={"ID":"b69094ee-26c9-4c0b-a6c5-042b65834156","Type":"ContainerStarted","Data":"cdfa8fd14f5b3822d738db1d7bf9fb9e8cca37daca89fd3198caec9ce3af1bee"}
Sep 18 19:38:45.611464 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.611439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6qv7" event={"ID":"a369c464-bec9-42e5-87ef-63208b3d90f6","Type":"ContainerStarted","Data":"e580a6cdab6fe6b5ade624ab6e38e1faf6fde5989510e3f0d2a64ffe5fe50444"}
Sep 18 19:38:45.613066 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.613043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vr48" event={"ID":"06f1c675-1a73-4385-864a-65d59dde6ad4","Type":"ContainerStarted","Data":"9578cae351b12ba2c3cf8329e4ed57dc72fa9135974495a3234bafc19ad8dfac"}
Sep 18 19:38:45.614714 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.614695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-768dq" event={"ID":"70ded459-00cc-4455-a8a4-a080de66c481","Type":"ContainerStarted","Data":"491e98cfd67c37eea3a4d26236d84cc402f1e89c85cb1591444ffd03d4f8406c"}
Sep 18 19:38:45.616838 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.616775    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-198.ec2.internal" podStartSLOduration=20.616763483 podStartE2EDuration="20.616763483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:29.58305987 +0000 UTC m=+7.362430453" watchObservedRunningTime="2026-09-18 19:38:45.616763483 +0000 UTC m=+23.396134064"
Sep 18 19:38:45.617281 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.617252    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dcpds" podStartSLOduration=3.977121533 podStartE2EDuration="21.617244804s" totalImagesPullingTime="17.640123271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:45.616640582 +0000 UTC m=+23.396011164" watchObservedRunningTime="2026-09-18 19:38:45.617244804 +0000 UTC m=+23.396615386"
Sep 18 19:38:45.629845 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.629810    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-768dq" podStartSLOduration=4.028205159 podStartE2EDuration="21.629798555s" totalImagesPullingTime="17.601593396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:45.629482298 +0000 UTC m=+23.408852881" watchObservedRunningTime="2026-09-18 19:38:45.629798555 +0000 UTC m=+23.409169137"
Sep 18 19:38:45.645868 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.645828    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6vr48" podStartSLOduration=3.9645987849999997 podStartE2EDuration="21.645817147s" totalImagesPullingTime="17.681218362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:45.645722461 +0000 UTC m=+23.425093044" watchObservedRunningTime="2026-09-18 19:38:45.645817147 +0000 UTC m=+23.425187721"
Sep 18 19:38:45.658644 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.658607    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b6qv7" podStartSLOduration=4.070302788 podStartE2EDuration="21.658595197s" totalImagesPullingTime="17.588292409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:45.658124741 +0000 UTC m=+23.437495323" watchObservedRunningTime="2026-09-18 19:38:45.658595197 +0000 UTC m=+23.437965779"
Sep 18 19:38:45.675529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.674177    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-csv52" podStartSLOduration=11.566165804 podStartE2EDuration="21.674159046s" totalImagesPullingTime="10.107993242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:45.673207789 +0000 UTC m=+23.452578372" watchObservedRunningTime="2026-09-18 19:38:45.674159046 +0000 UTC m=+23.453529628"
Sep 18 19:38:45.748448 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.748411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:45.749010 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.748990    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:45.801704 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.801554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:45.855550 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:45.855516    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:46.154691 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.154655    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:46.295230 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.295095    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:46.154674774Z","UUID":"909c0221-e600-48ea-8151-50f05a58923d","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:46.296901 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.296879    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:46.297032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.296910    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:46.348458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.348424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:46.348657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.348424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:46.348657 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:46.348560    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:46.348657 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:46.348646    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:46.623807 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.623723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"10cfb4a440be56c0132a509479fe9e33df0bc749099dcfaecf78417efcb2e2e8"}
Sep 18 19:38:46.626269 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:46.626234    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff" event={"ID":"b8030510-12d0-4d39-af1f-978983b22e86","Type":"ContainerStarted","Data":"104e191527cafc1b9f195cc4c00e4649929ca63875efc7bc2c486dc4ae87f0c9"}
Sep 18 19:38:47.347700 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.347669    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:47.347856 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:47.347786    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:47.630811 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.630726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff" event={"ID":"b8030510-12d0-4d39-af1f-978983b22e86","Type":"ContainerStarted","Data":"7d6d03cc92f378945cced8bdec94f0a2ca80dd18d3245aec57c1963b7cd6b4f4"}
Sep 18 19:38:47.632397 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.632359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5gtzj" event={"ID":"adcea9b5-b950-4d8b-a1e5-03c410160484","Type":"ContainerStarted","Data":"c08e2f5b29a4b7af3694b14a2f59e49aaad3a0ce25397fb52f341b78dfc91fd9"}
Sep 18 19:38:47.632509 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.632377    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:47.647752 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.647707    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bgrff" podStartSLOduration=4.597185696 podStartE2EDuration="23.647694085s" totalImagesPullingTime="19.050508389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:47.646547736 +0000 UTC m=+25.425918319" watchObservedRunningTime="2026-09-18 19:38:47.647694085 +0000 UTC m=+25.427064667"
Sep 18 19:38:47.660579 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.660527    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5gtzj" podStartSLOduration=6.072932354 podStartE2EDuration="23.660508359s" totalImagesPullingTime="17.587576005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:47.659872909 +0000 UTC m=+25.439243491" watchObservedRunningTime="2026-09-18 19:38:47.660508359 +0000 UTC m=+25.439878941"
Sep 18 19:38:47.791644 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.791621    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:47.893086 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:47.893007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:48.347582 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:48.347550    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:48.347582 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:48.347572    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:48.347794 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:48.347678    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:48.347794 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:48.347759    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:48.638815 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:48.638743    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"7bc9c597f016735c16ffbc42051b33f2a86551df84428a0d288116c3410c481e"}
Sep 18 19:38:49.347931 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:49.347898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:49.348095 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:49.348009    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:50.348432 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.348398    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:50.348833 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:50.348524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:50.348833 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.348589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:50.348833 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:50.348710    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:50.644948 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.644911    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="456228d1dfa8ad45a2a64238313f995f6aabcc5239b27cec08c6b72829172d1e" exitCode=0
Sep 18 19:38:50.645107 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.645006    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"456228d1dfa8ad45a2a64238313f995f6aabcc5239b27cec08c6b72829172d1e"}
Sep 18 19:38:50.648925 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.648899    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" event={"ID":"991a6b3a-db32-4391-ab67-00f063ea5999","Type":"ContainerStarted","Data":"ce5cceccbcc000f3a0f79329f91de0caf26a9e614655f8cdcf4dea058d5041c9"}
Sep 18 19:38:50.649183 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.649158    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:50.649263 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.649223    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:50.649263 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.649254    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:50.664547 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.664511    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:50.665011 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.664995    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:38:50.688274 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.688232    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9" podStartSLOduration=8.983261844 podStartE2EDuration="26.688218142s" totalImagesPullingTime="17.704956298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:50.68786865 +0000 UTC m=+28.467239230" watchObservedRunningTime="2026-09-18 19:38:50.688218142 +0000 UTC m=+28.467588723"
Sep 18 19:38:50.968095 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.968007    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:50.968250 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.968180    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:38:50.968558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:50.968537    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-csv52"
Sep 18 19:38:51.086542 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:51.086515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:51.348278 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:51.348235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:51.348483 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:51.348387    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:52.277618 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.277580    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s98hl"]
Sep 18 19:38:52.277771 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.277686    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:52.277823 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:52.277795    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:52.280624 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.280590    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6p9z"]
Sep 18 19:38:52.280760 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.280704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:52.280833 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:52.280809    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:52.281232 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.281211    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5qnvn"]
Sep 18 19:38:52.281331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.281281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:52.281392 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:52.281351    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:52.655067 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.655034    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="c3a0030837c8c995b737054a8864d078bf9d41abf365943564c3574d9d75cb78" exitCode=0
Sep 18 19:38:52.655943 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:52.655112    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"c3a0030837c8c995b737054a8864d078bf9d41abf365943564c3574d9d75cb78"}
Sep 18 19:38:54.349250 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:54.349076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:54.349645 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:54.349142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:54.349645 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:54.349310    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:54.349645 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:54.349407    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:54.349645 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:54.349170    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:54.349645 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:54.349485    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:54.662327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:54.662213    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="bb035ed60c866a7213606839088e208d029340a4a14c42161483e77442915ac7" exitCode=0
Sep 18 19:38:54.662327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:54.662267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"bb035ed60c866a7213606839088e208d029340a4a14c42161483e77442915ac7"}
Sep 18 19:38:56.348487 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:56.348447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:56.348487 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:56.348441    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:56.349004 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:56.348447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:56.349004 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:56.348597    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:38:56.349004 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:56.348755    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5qnvn" podUID="48cc6b9c-9b06-46ae-bbe7-2535006c2b4c"
Sep 18 19:38:56.349004 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:56.348837    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s98hl" podUID="aaab3569-3c45-4a03-bcd2-81b2b0a4f309"
Sep 18 19:38:57.955346 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:57.955314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:57.955694 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:57.955452    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:57.955694 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:57.955512    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret podName:aaab3569-3c45-4a03-bcd2-81b2b0a4f309 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:29.955497181 +0000 UTC m=+67.734867747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret") pod "global-pull-secret-syncer-s98hl" (UID: "aaab3569-3c45-4a03-bcd2-81b2b0a4f309") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:58.056318 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.056280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:58.056498 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.056417    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:58.056498 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.056475    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.056459565 +0000 UTC m=+67.835830125 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:58.157280 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.157246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:58.157467 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.157425    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:58.157467 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.157448    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:58.157467 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.157458    2567 projected.go:196] Error preparing data for projected volume kube-api-access-9l2zd for pod openshift-network-diagnostics/network-check-target-5qnvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:58.157629 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.157523    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd podName:48cc6b9c-9b06-46ae-bbe7-2535006c2b4c nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.157500201 +0000 UTC m=+67.936870802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l2zd" (UniqueName: "kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd") pod "network-check-target-5qnvn" (UID: "48cc6b9c-9b06-46ae-bbe7-2535006c2b4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:58.201172 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.201138    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-198.ec2.internal" event="NodeReady"
Sep 18 19:38:58.201319 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.201268    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:38:58.251818 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.251734    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757975c5df-ltrdc"]
Sep 18 19:38:58.283851 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.283821    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7bsqr"]
Sep 18 19:38:58.284027 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.283939    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.286594 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.286566    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:38:58.286896 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.286878    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rwqd5\""
Sep 18 19:38:58.287126 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.287100    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:38:58.287541 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.287512    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:38:58.296051 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.296018    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:38:58.300839 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.300810    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757975c5df-ltrdc"]
Sep 18 19:38:58.300839 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.300853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7bsqr"]
Sep 18 19:38:58.301023 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.300880    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-djj5j"]
Sep 18 19:38:58.301023 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.300945    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.304556 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.304535    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:38:58.304674 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.304591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:38:58.304929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.304905    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j9gtw\""
Sep 18 19:38:58.318782 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.318682    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djj5j"]
Sep 18 19:38:58.318911 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.318801    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.321816 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.321781    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:38:58.321934 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.321894    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:38:58.321979 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.321786    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:38:58.322208 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.322188    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kvwkp\""
Sep 18 19:38:58.348082 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.348049    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:38:58.348259 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.348049    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:38:58.348259 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.348186    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:38:58.350505 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350479    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:38:58.350641 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350490    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:38:58.350929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350815    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nndrn\""
Sep 18 19:38:58.350929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:38:58.350929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350891    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:38:58.351684 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.350988    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nv7t\""
Sep 18 19:38:58.359357 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359357 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359710 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359552    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.359710 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.359592    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nn4d\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460122 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460122 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pmq\" (UniqueName: \"kubernetes.io/projected/11bdee4d-4727-49bd-91e4-8ba898621b9c-kube-api-access-49pmq\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.460345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nn4d\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f430df-4b07-4547-b71e-68541a5134ab-config-volume\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.460345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460358    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f430df-4b07-4547-b71e-68541a5134ab-tmp-dir\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sk7r\" (UniqueName: \"kubernetes.io/projected/06f430df-4b07-4547-b71e-68541a5134ab-kube-api-access-9sk7r\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.460402    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.460478    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460546    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bdee4d-4727-49bd-91e4-8ba898621b9c-cert\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.460860 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.460583    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.960561812 +0000 UTC m=+36.739932371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:38:58.460860 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460964 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.460964 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.460959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.461376 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.461341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.461624 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.461602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.465539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.465492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.465668 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.465609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.471445 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.471424    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.471560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.471540    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nn4d\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.561589 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f430df-4b07-4547-b71e-68541a5134ab-tmp-dir\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.561589 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sk7r\" (UniqueName: \"kubernetes.io/projected/06f430df-4b07-4547-b71e-68541a5134ab-kube-api-access-9sk7r\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.561842 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bdee4d-4727-49bd-91e4-8ba898621b9c-cert\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.561842 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.561842 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pmq\" (UniqueName: \"kubernetes.io/projected/11bdee4d-4727-49bd-91e4-8ba898621b9c-kube-api-access-49pmq\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.561842 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f430df-4b07-4547-b71e-68541a5134ab-config-volume\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.561842 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.561829    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:58.562065 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.561900    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:38:59.061878652 +0000 UTC m=+36.841249228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:58.562065 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.561975    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f430df-4b07-4547-b71e-68541a5134ab-tmp-dir\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.562323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.562295    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f430df-4b07-4547-b71e-68541a5134ab-config-volume\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.564988 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.564964    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11bdee4d-4727-49bd-91e4-8ba898621b9c-cert\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.573798 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.573779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sk7r\" (UniqueName: \"kubernetes.io/projected/06f430df-4b07-4547-b71e-68541a5134ab-kube-api-access-9sk7r\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:58.575395 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.575360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pmq\" (UniqueName: \"kubernetes.io/projected/11bdee4d-4727-49bd-91e4-8ba898621b9c-kube-api-access-49pmq\") pod \"ingress-canary-djj5j\" (UID: \"11bdee4d-4727-49bd-91e4-8ba898621b9c\") " pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.628345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.628314    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djj5j"
Sep 18 19:38:58.965638 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:58.965545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:58.965960 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.965654    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:58.965960 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.965673    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:38:58.965960 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:58.965730    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:59.965715838 +0000 UTC m=+37.745086398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:38:59.012450 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.012421    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djj5j"]
Sep 18 19:38:59.030203 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:38:59.030175    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bdee4d_4727_49bd_91e4_8ba898621b9c.slice/crio-e907d2c2c05ef19c53bc5ec55414395c7e8537a584e0895843907ef9415a3b84 WatchSource:0}: Error finding container e907d2c2c05ef19c53bc5ec55414395c7e8537a584e0895843907ef9415a3b84: Status 404 returned error can't find the container with id e907d2c2c05ef19c53bc5ec55414395c7e8537a584e0895843907ef9415a3b84
Sep 18 19:38:59.066076 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.066055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:38:59.066216 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:59.066201    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:59.066281 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:59.066266    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.066246592 +0000 UTC m=+37.845617157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:59.675386 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.675187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djj5j" event={"ID":"11bdee4d-4727-49bd-91e4-8ba898621b9c","Type":"ContainerStarted","Data":"e907d2c2c05ef19c53bc5ec55414395c7e8537a584e0895843907ef9415a3b84"}
Sep 18 19:38:59.677743 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.677718    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="eb6a399ac77034f1448ded5a35822dbee0698faa9ed2d12955e56b6c9333f5c4" exitCode=0
Sep 18 19:38:59.677846 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.677753    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"eb6a399ac77034f1448ded5a35822dbee0698faa9ed2d12955e56b6c9333f5c4"}
Sep 18 19:38:59.973671 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:38:59.973581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:38:59.974061 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:59.973749    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:59.974061 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:59.973765    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:38:59.974061 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:38:59.973832    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:01.973813421 +0000 UTC m=+39.753183997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:39:00.074105 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:00.074071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:39:00.074283 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:00.074255    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:00.074375 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:00.074349    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.074327206 +0000 UTC m=+39.853697765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:00.684255 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:00.684219    2567 generic.go:356] "Generic (PLEG): container finished" podID="1608a4d9-2d53-4b44-b6f1-b7159dce3a8f" containerID="d62ebe83d1b27bf9d9d5c7dbc9e7bb47303086887fa6c76c131ccc54ce338de6" exitCode=0
Sep 18 19:39:00.684499 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:00.684299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerDied","Data":"d62ebe83d1b27bf9d9d5c7dbc9e7bb47303086887fa6c76c131ccc54ce338de6"}
Sep 18 19:39:01.690288 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:01.690253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfw" event={"ID":"1608a4d9-2d53-4b44-b6f1-b7159dce3a8f","Type":"ContainerStarted","Data":"0d85396dd8253adace62838a57296b92324389529ce1319305c52cb50fb3d732"}
Sep 18 19:39:01.708853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:01.708804    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-27sfw" podStartSLOduration=9.171749063 podStartE2EDuration="37.708787235s" totalImagesPullingTime="28.537038172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:01.708031657 +0000 UTC m=+39.487402256" watchObservedRunningTime="2026-09-18 19:39:01.708787235 +0000 UTC m=+39.488157817"
Sep 18 19:39:01.990833 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:01.990804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:39:01.990984 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:01.990963    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:01.991024 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:01.990986    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:39:01.991058 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:01.991040    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.991023667 +0000 UTC m=+43.770394227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:39:02.091540 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:02.091503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:39:02.091691 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:02.091661    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:02.091846 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:02.091726    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:39:06.091711447 +0000 UTC m=+43.871082006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:02.694610 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:02.694577    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djj5j" event={"ID":"11bdee4d-4727-49bd-91e4-8ba898621b9c","Type":"ContainerStarted","Data":"25be713462d080b86ea20185474e06da435a08d6d8d0cf0ad29475860bef2aa5"}
Sep 18 19:39:02.710460 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:02.710413    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-djj5j" podStartSLOduration=2.128475914 podStartE2EDuration="4.710397217s" totalImagesPullingTime="2.581921303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:02.709084443 +0000 UTC m=+40.488455025" watchObservedRunningTime="2026-09-18 19:39:02.710397217 +0000 UTC m=+40.489767798"
Sep 18 19:39:06.020021 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:06.019984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:39:06.020434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:06.020110    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:06.020434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:06.020122    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:39:06.020434 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:06.020170    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:14.020156479 +0000 UTC m=+51.799527039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:39:06.120427 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:06.120386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:39:06.120561 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:06.120518    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:06.120597 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:06.120590    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:39:14.120575641 +0000 UTC m=+51.899946201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:14.075349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:14.075302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:39:14.075842 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:14.075437    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:14.075842 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:14.075450    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:39:14.075842 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:14.075500    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.075486079 +0000 UTC m=+67.854856638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:39:14.176082 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:14.176056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:39:14.176195 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:14.176161    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:14.176232 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:14.176208    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:39:30.176195298 +0000 UTC m=+67.955565859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:22.672757 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:22.672727    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7nd9"
Sep 18 19:39:29.983412 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:29.983382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:39:29.985515 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:29.985497    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:29.996898 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:29.996873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aaab3569-3c45-4a03-bcd2-81b2b0a4f309-original-pull-secret\") pod \"global-pull-secret-syncer-s98hl\" (UID: \"aaab3569-3c45-4a03-bcd2-81b2b0a4f309\") " pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:39:30.084247 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.084223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:39:30.084341 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.084252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:39:30.084410 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.084353    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:30.084410 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.084377    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:39:30.084484 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.084433    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:40:02.084419845 +0000 UTC m=+99.863790406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:39:30.086045 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.086030    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:30.094459 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.094441    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:39:30.094507 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.094487    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:40:34.094474975 +0000 UTC m=+131.873845535 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : secret "metrics-daemon-secret" not found
Sep 18 19:39:30.165775 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.165748    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s98hl"
Sep 18 19:39:30.185322 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.185290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:39:30.185461 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.185327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:39:30.185461 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.185447    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:30.185576 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:39:30.185505    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:40:02.185490978 +0000 UTC m=+99.964861541 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:30.187525 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.187501    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:30.197718 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.197699    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:30.209486 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.209466    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2zd\" (UniqueName: \"kubernetes.io/projected/48cc6b9c-9b06-46ae-bbe7-2535006c2b4c-kube-api-access-9l2zd\") pod \"network-check-target-5qnvn\" (UID: \"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c\") " pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:39:30.435253 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.435214    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s98hl"]
Sep 18 19:39:30.439434 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:39:30.439412    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaab3569_3c45_4a03_bcd2_81b2b0a4f309.slice/crio-10fba839ffbcfa0c4d91444a92175c795ad42cb2bef165e5057ca8f4b38edb89 WatchSource:0}: Error finding container 10fba839ffbcfa0c4d91444a92175c795ad42cb2bef165e5057ca8f4b38edb89: Status 404 returned error can't find the container with id 10fba839ffbcfa0c4d91444a92175c795ad42cb2bef165e5057ca8f4b38edb89
Sep 18 19:39:30.461690 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.461667    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nndrn\""
Sep 18 19:39:30.470173 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.470156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:39:30.736270 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.736187    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5qnvn"]
Sep 18 19:39:30.740178 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:39:30.740151    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cc6b9c_9b06_46ae_bbe7_2535006c2b4c.slice/crio-72935827fb54dc7700ce6fb12056a8d94cb395665c5bc7809674d95cea1c2b72 WatchSource:0}: Error finding container 72935827fb54dc7700ce6fb12056a8d94cb395665c5bc7809674d95cea1c2b72: Status 404 returned error can't find the container with id 72935827fb54dc7700ce6fb12056a8d94cb395665c5bc7809674d95cea1c2b72
Sep 18 19:39:30.756940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.756915    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5qnvn" event={"ID":"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c","Type":"ContainerStarted","Data":"72935827fb54dc7700ce6fb12056a8d94cb395665c5bc7809674d95cea1c2b72"}
Sep 18 19:39:30.758049 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:30.758024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s98hl" event={"ID":"aaab3569-3c45-4a03-bcd2-81b2b0a4f309","Type":"ContainerStarted","Data":"10fba839ffbcfa0c4d91444a92175c795ad42cb2bef165e5057ca8f4b38edb89"}
Sep 18 19:39:35.772208 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:35.772168    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5qnvn" event={"ID":"48cc6b9c-9b06-46ae-bbe7-2535006c2b4c","Type":"ContainerStarted","Data":"dfcf8d9d5c3abe3f37d6fcafb41c41fd4512a55d364fe80ca3803274947a00a4"}
Sep 18 19:39:35.772691 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:35.772261    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:39:35.773606 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:35.773576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s98hl" event={"ID":"aaab3569-3c45-4a03-bcd2-81b2b0a4f309","Type":"ContainerStarted","Data":"96435e93c30eddffb32d13f1c36bb6d5fcdf341eb6b7d7e215b778348181afe9"}
Sep 18 19:39:35.784742 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:35.784690    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5qnvn" podStartSLOduration=67.652323317 podStartE2EDuration="1m11.7846765s" totalImagesPullingTime="4.132353183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:35.784587334 +0000 UTC m=+73.563957915" watchObservedRunningTime="2026-09-18 19:39:35.7846765 +0000 UTC m=+73.564047075"
Sep 18 19:39:35.796615 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:39:35.796575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s98hl" podStartSLOduration=67.358851477 podStartE2EDuration="1m11.796541493s" totalImagesPullingTime="4.437690016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:35.796181038 +0000 UTC m=+73.575551622" watchObservedRunningTime="2026-09-18 19:39:35.796541493 +0000 UTC m=+73.575912075"
Sep 18 19:40:02.101169 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:40:02.101125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:40:02.101679 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:02.101298    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:40:02.101679 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:02.101322    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:40:02.101679 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:02.101429    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:41:06.101409215 +0000 UTC m=+163.880779788 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:40:02.201733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:40:02.201701    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:40:02.201927 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:02.201826    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:02.201927 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:02.201891    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:41:06.201871594 +0000 UTC m=+163.981242161 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:40:06.779213 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:40:06.779183    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5qnvn"
Sep 18 19:40:34.118299 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:40:34.118244    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:40:34.118748 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:34.118406    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:40:34.118748 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:40:34.118487    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:42:36.118471009 +0000 UTC m=+253.897841568 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : secret "metrics-daemon-secret" not found
Sep 18 19:41:01.299486 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:01.299447    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-757975c5df-ltrdc" podUID="fca47ba9-859f-45b0-9285-0614e5101f21"
Sep 18 19:41:01.311631 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:01.311576    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7bsqr" podUID="06f430df-4b07-4547-b71e-68541a5134ab"
Sep 18 19:41:01.371536 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:01.371485    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:41:01.977558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:41:01.977480    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:41:01.977692 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:41:01.977480    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:41:06.148685 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:41:06.148641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") pod \"image-registry-757975c5df-ltrdc\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") " pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:41:06.149194 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:06.148790    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:41:06.149194 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:06.148810    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757975c5df-ltrdc: secret "image-registry-tls" not found
Sep 18 19:41:06.149194 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:06.148885    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls podName:fca47ba9-859f-45b0-9285-0614e5101f21 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:08.148863472 +0000 UTC m=+285.928234032 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls") pod "image-registry-757975c5df-ltrdc" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21") : secret "image-registry-tls" not found
Sep 18 19:41:06.249218 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:41:06.249122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:41:06.249357 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:06.249278    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:41:06.249357 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:41:06.249349    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls podName:06f430df-4b07-4547-b71e-68541a5134ab nodeName:}" failed. No retries permitted until 2026-09-18 19:43:08.249333701 +0000 UTC m=+286.028704260 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls") pod "dns-default-7bsqr" (UID: "06f430df-4b07-4547-b71e-68541a5134ab") : secret "dns-default-metrics-tls" not found
Sep 18 19:41:12.348170 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:41:12.348131    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:42:22.919015 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.918977    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"]
Sep 18 19:42:22.921808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.921791    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"]
Sep 18 19:42:22.921929 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.921909    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:22.923972 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.923950    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:22.924804 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.924782    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 19:42:22.924972 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.924795    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 19:42:22.925281 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.924959    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:22.925739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.925579    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 19:42:22.925739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.925688    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 19:42:22.925874 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.925738    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jjx64\""
Sep 18 19:42:22.926012 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.925997    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4vsvd"]
Sep 18 19:42:22.928032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.928016    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"]
Sep 18 19:42:22.928143 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.928130    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:22.929513 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.929492    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 19:42:22.929678 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.929665    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:22.930021 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930001    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 19:42:22.930110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-v27rw\""
Sep 18 19:42:22.930164 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930105    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:42:22.930218 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930194    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"]
Sep 18 19:42:22.930292 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930271    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"
Sep 18 19:42:22.930441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930424    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:42:22.930531 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930202    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-chflr\""
Sep 18 19:42:22.930592 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930404    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:42:22.930644 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.930211    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:42:22.932529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.932509    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"
Sep 18 19:42:22.934796 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.934759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:22.937707 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.937690    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"]
Sep 18 19:42:22.942587 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.941396    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:42:22.942587 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.942265    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qtjsp\""
Sep 18 19:42:22.942860 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.942673    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"]
Sep 18 19:42:22.942860 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.942691    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:42:22.942964 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.942867    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cfghk\""
Sep 18 19:42:22.943930 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.943914    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"]
Sep 18 19:42:22.944755 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.944738    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4vsvd"]
Sep 18 19:42:22.961194 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:22.961174    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"]
Sep 18 19:42:23.014595 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.014564    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"]
Sep 18 19:42:23.016851 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.016654    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.022462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.020319    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:42:23.022462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.020579    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:42:23.022462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.021443    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-96pt7\""
Sep 18 19:42:23.022462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.021956    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"]
Sep 18 19:42:23.022462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.022215    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:42:23.022961 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.022944    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:42:23.025004 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.024987    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-gkkdr"]
Sep 18 19:42:23.025121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.025105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.027053 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.027036    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.028209 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.027710    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:23.028209 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.027931    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:42:23.028209 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.027710    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:42:23.028209 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.028073    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jqhtz\""
Sep 18 19:42:23.028753 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.028737    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"]
Sep 18 19:42:23.028851 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.028777    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 19:42:23.028915 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.028852    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d6srl\""
Sep 18 19:42:23.028964 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.028910    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 19:42:23.029208 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.029192    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:23.029284 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.029268    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 19:42:23.034753 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.034733    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-gkkdr"]
Sep 18 19:42:23.034888 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.034869    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 19:42:23.035832 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.035814    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"]
Sep 18 19:42:23.062142 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnst\" (UniqueName: \"kubernetes.io/projected/aa6278a5-600b-45eb-a49e-efbadf6d76b4-kube-api-access-chnst\") pod \"network-check-source-7cbc45fddc-hwh7h\" (UID: \"aa6278a5-600b-45eb-a49e-efbadf6d76b4\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"
Sep 18 19:42:23.062258 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vmm\" (UniqueName: \"kubernetes.io/projected/051ad5ae-9e24-48fa-940d-fe0ea43a9491-kube-api-access-x9vmm\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.062258 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.062258 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd8f8e0-03ee-4342-b072-5d602b5a60be-serving-cert\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.062258 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx9g\" (UniqueName: \"kubernetes.io/projected/6cd8f8e0-03ee-4342-b072-5d602b5a60be-kube-api-access-ntx9g\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/051ad5ae-9e24-48fa-940d-fe0ea43a9491-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062308    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/051ad5ae-9e24-48fa-940d-fe0ea43a9491-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6lc\" (UniqueName: \"kubernetes.io/projected/093c8031-69fd-4cde-84f0-cdd7614bdb8b-kube-api-access-vf6lc\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd8f8e0-03ee-4342-b072-5d602b5a60be-config\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.062442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093c8031-69fd-4cde-84f0-cdd7614bdb8b-serving-cert\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.062613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvhq\" (UniqueName: \"kubernetes.io/projected/b6b0fcff-6aef-482c-a3ff-05e28e0cf311-kube-api-access-hgvhq\") pod \"volume-data-source-validator-5c4db8bdf9-mcgs7\" (UID: \"b6b0fcff-6aef-482c-a3ff-05e28e0cf311\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"
Sep 18 19:42:23.062613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-snapshots\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.062613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.062497    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-tmp\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.163567 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093c8031-69fd-4cde-84f0-cdd7614bdb8b-serving-cert\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.163567 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-trusted-ca\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.163567 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93f8637f-13ec-4134-ab92-45d02e00ce46-serving-cert\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.163878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163607    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-snapshots\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.163878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.163878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-tmp\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.163878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163840    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnnn\" (UniqueName: \"kubernetes.io/projected/93d22068-2c02-49e6-b708-69d35f922a48-kube-api-access-lhnnn\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.163878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vmm\" (UniqueName: \"kubernetes.io/projected/051ad5ae-9e24-48fa-940d-fe0ea43a9491-kube-api-access-x9vmm\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.164095 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnst\" (UniqueName: \"kubernetes.io/projected/aa6278a5-600b-45eb-a49e-efbadf6d76b4-kube-api-access-chnst\") pod \"network-check-source-7cbc45fddc-hwh7h\" (UID: \"aa6278a5-600b-45eb-a49e-efbadf6d76b4\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"
Sep 18 19:42:23.164095 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.164095 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.163976    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a72572f6-da61-48e6-98cd-ab9d4478cd23-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.164095 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164008    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/051ad5ae-9e24-48fa-940d-fe0ea43a9491-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.164285 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.164285 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crj5\" (UniqueName: \"kubernetes.io/projected/a72572f6-da61-48e6-98cd-ab9d4478cd23-kube-api-access-6crj5\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.164285 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvhq\" (UniqueName: \"kubernetes.io/projected/b6b0fcff-6aef-482c-a3ff-05e28e0cf311-kube-api-access-hgvhq\") pod \"volume-data-source-validator-5c4db8bdf9-mcgs7\" (UID: \"b6b0fcff-6aef-482c-a3ff-05e28e0cf311\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"
Sep 18 19:42:23.164285 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164243    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.164285 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6lc\" (UniqueName: \"kubernetes.io/projected/093c8031-69fd-4cde-84f0-cdd7614bdb8b-kube-api-access-vf6lc\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-config\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkjl\" (UniqueName: \"kubernetes.io/projected/93f8637f-13ec-4134-ab92-45d02e00ce46-kube-api-access-mxkjl\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd8f8e0-03ee-4342-b072-5d602b5a60be-serving-cert\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx9g\" (UniqueName: \"kubernetes.io/projected/6cd8f8e0-03ee-4342-b072-5d602b5a60be-kube-api-access-ntx9g\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/051ad5ae-9e24-48fa-940d-fe0ea43a9491-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-snapshots\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.164529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164466    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd8f8e0-03ee-4342-b072-5d602b5a60be-config\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.164849 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/051ad5ae-9e24-48fa-940d-fe0ea43a9491-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.164849 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.164949 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.164883    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/093c8031-69fd-4cde-84f0-cdd7614bdb8b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.165053 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.165019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cd8f8e0-03ee-4342-b072-5d602b5a60be-config\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.165206 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.165140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/093c8031-69fd-4cde-84f0-cdd7614bdb8b-tmp\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.166417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.166352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/093c8031-69fd-4cde-84f0-cdd7614bdb8b-serving-cert\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.167159 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.167138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cd8f8e0-03ee-4342-b072-5d602b5a60be-serving-cert\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.167234 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.167188    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/051ad5ae-9e24-48fa-940d-fe0ea43a9491-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.171606 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.171552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vmm\" (UniqueName: \"kubernetes.io/projected/051ad5ae-9e24-48fa-940d-fe0ea43a9491-kube-api-access-x9vmm\") pod \"kube-storage-version-migrator-operator-55889f7b47-csqxv\" (UID: \"051ad5ae-9e24-48fa-940d-fe0ea43a9491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.172125 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.172077    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx9g\" (UniqueName: \"kubernetes.io/projected/6cd8f8e0-03ee-4342-b072-5d602b5a60be-kube-api-access-ntx9g\") pod \"service-ca-operator-57cdc4b768-zl9p5\" (UID: \"6cd8f8e0-03ee-4342-b072-5d602b5a60be\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.172288 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.172262    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6lc\" (UniqueName: \"kubernetes.io/projected/093c8031-69fd-4cde-84f0-cdd7614bdb8b-kube-api-access-vf6lc\") pod \"insights-operator-6f9c644dd9-4vsvd\" (UID: \"093c8031-69fd-4cde-84f0-cdd7614bdb8b\") " pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.172470 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.172269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvhq\" (UniqueName: \"kubernetes.io/projected/b6b0fcff-6aef-482c-a3ff-05e28e0cf311-kube-api-access-hgvhq\") pod \"volume-data-source-validator-5c4db8bdf9-mcgs7\" (UID: \"b6b0fcff-6aef-482c-a3ff-05e28e0cf311\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"
Sep 18 19:42:23.172470 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.172433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnst\" (UniqueName: \"kubernetes.io/projected/aa6278a5-600b-45eb-a49e-efbadf6d76b4-kube-api-access-chnst\") pod \"network-check-source-7cbc45fddc-hwh7h\" (UID: \"aa6278a5-600b-45eb-a49e-efbadf6d76b4\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"
Sep 18 19:42:23.233165 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.233122    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"
Sep 18 19:42:23.239808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.239791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"
Sep 18 19:42:23.246730 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.246704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd"
Sep 18 19:42:23.254016 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.254000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"
Sep 18 19:42:23.259401 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.259157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"
Sep 18 19:42:23.265841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.265841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-config\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.265841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkjl\" (UniqueName: \"kubernetes.io/projected/93f8637f-13ec-4134-ab92-45d02e00ce46-kube-api-access-mxkjl\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.265841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-trusted-ca\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93f8637f-13ec-4134-ab92-45d02e00ce46-serving-cert\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.265873    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.265924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnnn\" (UniqueName: \"kubernetes.io/projected/93d22068-2c02-49e6-b708-69d35f922a48-kube-api-access-lhnnn\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.265940    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls podName:93d22068-2c02-49e6-b708-69d35f922a48 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:23.765920793 +0000 UTC m=+241.545291376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8q6vw" (UID: "93d22068-2c02-49e6-b708-69d35f922a48") : secret "samples-operator-tls" not found
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.266002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a72572f6-da61-48e6-98cd-ab9d4478cd23-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.266110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.266039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crj5\" (UniqueName: \"kubernetes.io/projected/a72572f6-da61-48e6-98cd-ab9d4478cd23-kube-api-access-6crj5\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.266559 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.266476    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:23.266559 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.266522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-config\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.266559 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.266537    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:23.766522019 +0000 UTC m=+241.545892584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:23.266953 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.266908    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a72572f6-da61-48e6-98cd-ab9d4478cd23-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.267251 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.267228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93f8637f-13ec-4134-ab92-45d02e00ce46-trusted-ca\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.269259 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.269238    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93f8637f-13ec-4134-ab92-45d02e00ce46-serving-cert\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.275626 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.275601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crj5\" (UniqueName: \"kubernetes.io/projected/a72572f6-da61-48e6-98cd-ab9d4478cd23-kube-api-access-6crj5\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.278482 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.278422    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkjl\" (UniqueName: \"kubernetes.io/projected/93f8637f-13ec-4134-ab92-45d02e00ce46-kube-api-access-mxkjl\") pod \"console-operator-fd4bb46c-gkkdr\" (UID: \"93f8637f-13ec-4134-ab92-45d02e00ce46\") " pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.279615 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.279560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnnn\" (UniqueName: \"kubernetes.io/projected/93d22068-2c02-49e6-b708-69d35f922a48-kube-api-access-lhnnn\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.344075 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.343946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:23.515173 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.515102    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5"]
Sep 18 19:42:23.518209 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:23.518179    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd8f8e0_03ee_4342_b072_5d602b5a60be.slice/crio-3dc81026135c8081582ecf0036dec1683f7d531ac9d0b420350ead28875013a9 WatchSource:0}: Error finding container 3dc81026135c8081582ecf0036dec1683f7d531ac9d0b420350ead28875013a9: Status 404 returned error can't find the container with id 3dc81026135c8081582ecf0036dec1683f7d531ac9d0b420350ead28875013a9
Sep 18 19:42:23.518444 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.518424    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv"]
Sep 18 19:42:23.541939 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.541916    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7"]
Sep 18 19:42:23.544753 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:23.544726    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b0fcff_6aef_482c_a3ff_05e28e0cf311.slice/crio-a771e9f0a8e3eb7c6d597e5f07c40857d5143d3a797864da5cb4c5b66e56a466 WatchSource:0}: Error finding container a771e9f0a8e3eb7c6d597e5f07c40857d5143d3a797864da5cb4c5b66e56a466: Status 404 returned error can't find the container with id a771e9f0a8e3eb7c6d597e5f07c40857d5143d3a797864da5cb4c5b66e56a466
Sep 18 19:42:23.545781 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.545760    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4vsvd"]
Sep 18 19:42:23.547908 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:23.547886    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093c8031_69fd_4cde_84f0_cdd7614bdb8b.slice/crio-ac2ab37e890ecb56572c94a6de4e2a1282c05d0d8256b9db5cef0701b11af067 WatchSource:0}: Error finding container ac2ab37e890ecb56572c94a6de4e2a1282c05d0d8256b9db5cef0701b11af067: Status 404 returned error can't find the container with id ac2ab37e890ecb56572c94a6de4e2a1282c05d0d8256b9db5cef0701b11af067
Sep 18 19:42:23.563826 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.563806    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h"]
Sep 18 19:42:23.566283 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:23.566260    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6278a5_600b_45eb_a49e_efbadf6d76b4.slice/crio-63c3f8a083aaaeaf3ef95dbe3ea8e0972051824320247517196e4dbf5807eb21 WatchSource:0}: Error finding container 63c3f8a083aaaeaf3ef95dbe3ea8e0972051824320247517196e4dbf5807eb21: Status 404 returned error can't find the container with id 63c3f8a083aaaeaf3ef95dbe3ea8e0972051824320247517196e4dbf5807eb21
Sep 18 19:42:23.615145 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.615120    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-gkkdr"]
Sep 18 19:42:23.618235 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:23.618211    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f8637f_13ec_4134_ab92_45d02e00ce46.slice/crio-4462aa97708a11ddb6e9e4929ac8055abd72b43dde042556d4b76a0c9b253026 WatchSource:0}: Error finding container 4462aa97708a11ddb6e9e4929ac8055abd72b43dde042556d4b76a0c9b253026: Status 404 returned error can't find the container with id 4462aa97708a11ddb6e9e4929ac8055abd72b43dde042556d4b76a0c9b253026
Sep 18 19:42:23.769326 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.769237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:23.769326 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:23.769313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:23.769580 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.769423    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:23.769580 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.769494    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:24.769478083 +0000 UTC m=+242.548848642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:23.769580 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.769426    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:42:23.769580 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:23.769569    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls podName:93d22068-2c02-49e6-b708-69d35f922a48 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:24.769555934 +0000 UTC m=+242.548926495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8q6vw" (UID: "93d22068-2c02-49e6-b708-69d35f922a48") : secret "samples-operator-tls" not found
Sep 18 19:42:24.177630 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.177588    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv" event={"ID":"051ad5ae-9e24-48fa-940d-fe0ea43a9491","Type":"ContainerStarted","Data":"36a60e9d88030d7cde571e6efa075a7e8a530b4300c0759976e3b3dbdb060c35"}
Sep 18 19:42:24.183590 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.183560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7" event={"ID":"b6b0fcff-6aef-482c-a3ff-05e28e0cf311","Type":"ContainerStarted","Data":"a771e9f0a8e3eb7c6d597e5f07c40857d5143d3a797864da5cb4c5b66e56a466"}
Sep 18 19:42:24.185823 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.185737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" event={"ID":"93f8637f-13ec-4134-ab92-45d02e00ce46","Type":"ContainerStarted","Data":"4462aa97708a11ddb6e9e4929ac8055abd72b43dde042556d4b76a0c9b253026"}
Sep 18 19:42:24.188494 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.188472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h" event={"ID":"aa6278a5-600b-45eb-a49e-efbadf6d76b4","Type":"ContainerStarted","Data":"e3e75240663740949cd2d71efa3c4ad31d6fd6ce1e3e28ef6f0da1361b9fdb6e"}
Sep 18 19:42:24.188607 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.188512    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h" event={"ID":"aa6278a5-600b-45eb-a49e-efbadf6d76b4","Type":"ContainerStarted","Data":"63c3f8a083aaaeaf3ef95dbe3ea8e0972051824320247517196e4dbf5807eb21"}
Sep 18 19:42:24.190638 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.190614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd" event={"ID":"093c8031-69fd-4cde-84f0-cdd7614bdb8b","Type":"ContainerStarted","Data":"ac2ab37e890ecb56572c94a6de4e2a1282c05d0d8256b9db5cef0701b11af067"}
Sep 18 19:42:24.195911 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.194424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5" event={"ID":"6cd8f8e0-03ee-4342-b072-5d602b5a60be","Type":"ContainerStarted","Data":"3dc81026135c8081582ecf0036dec1683f7d531ac9d0b420350ead28875013a9"}
Sep 18 19:42:24.204649 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.204329    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hwh7h" podStartSLOduration=2.20431248 podStartE2EDuration="2.20431248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:24.202689982 +0000 UTC m=+241.982060565" watchObservedRunningTime="2026-09-18 19:42:24.20431248 +0000 UTC m=+241.983683057"
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.779019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:24.779095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:24.779145    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:24.779221    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls podName:93d22068-2c02-49e6-b708-69d35f922a48 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:26.779198988 +0000 UTC m=+244.558569549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8q6vw" (UID: "93d22068-2c02-49e6-b708-69d35f922a48") : secret "samples-operator-tls" not found
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:24.779231    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:24.779306 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:24.779280    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:26.77926545 +0000 UTC m=+244.558636016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:26.202869 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:26.202834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7" event={"ID":"b6b0fcff-6aef-482c-a3ff-05e28e0cf311","Type":"ContainerStarted","Data":"4ae41ab717109f0fa37c2e445f136f4e5a3fb7179a4bdf6d994499f994a722a7"}
Sep 18 19:42:26.217153 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:26.217101    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-mcgs7" podStartSLOduration=2.621530968 podStartE2EDuration="4.217082842s" totalImagesPullingTime="1.595551874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:26.216201686 +0000 UTC m=+243.995572281" watchObservedRunningTime="2026-09-18 19:42:26.217082842 +0000 UTC m=+243.996453423"
Sep 18 19:42:26.797553 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:26.797508    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:26.797739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:26.797589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:26.797739 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:26.797663    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:42:26.797739 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:26.797734    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls podName:93d22068-2c02-49e6-b708-69d35f922a48 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:30.797716611 +0000 UTC m=+248.577087194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8q6vw" (UID: "93d22068-2c02-49e6-b708-69d35f922a48") : secret "samples-operator-tls" not found
Sep 18 19:42:26.797924 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:26.797739    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:26.797924 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:26.797825    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:30.797806199 +0000 UTC m=+248.577176776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:28.145512 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.145479    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lth96"]
Sep 18 19:42:28.147668 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.147652    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.149497 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.149476    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:42:28.149634 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.149537    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:42:28.149754 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.149741    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvsph\""
Sep 18 19:42:28.159920 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.159898    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lth96"]
Sep 18 19:42:28.214549 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.214522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/0.log"
Sep 18 19:42:28.214720 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.214569    2567 generic.go:356] "Generic (PLEG): container finished" podID="93f8637f-13ec-4134-ab92-45d02e00ce46" containerID="40ed196ff9c49325b5a295de41ea8bb43f026321b8dd8b4f15f18fc4189f418f" exitCode=255
Sep 18 19:42:28.214720 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.214657    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" event={"ID":"93f8637f-13ec-4134-ab92-45d02e00ce46","Type":"ContainerDied","Data":"40ed196ff9c49325b5a295de41ea8bb43f026321b8dd8b4f15f18fc4189f418f"}
Sep 18 19:42:28.214891 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.214874    2567 scope.go:117] "RemoveContainer" containerID="40ed196ff9c49325b5a295de41ea8bb43f026321b8dd8b4f15f18fc4189f418f"
Sep 18 19:42:28.216382 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.216340    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd" event={"ID":"093c8031-69fd-4cde-84f0-cdd7614bdb8b","Type":"ContainerStarted","Data":"ebee031f8723baed5715b4bfe9741788b555780f44dbd3f53e99d5c366d23303"}
Sep 18 19:42:28.218085 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.218066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5" event={"ID":"6cd8f8e0-03ee-4342-b072-5d602b5a60be","Type":"ContainerStarted","Data":"eaf04bcda04e6e694d2d2c25ec977728e6a1c9a8837e0fa3bc1b4da9b6836b7f"}
Sep 18 19:42:28.219779 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.219750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv" event={"ID":"051ad5ae-9e24-48fa-940d-fe0ea43a9491","Type":"ContainerStarted","Data":"78e514a0a562b83c97896236e19473ed74d164814bbc573829e6bb2b8f418f58"}
Sep 18 19:42:28.242464 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.242415    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5" podStartSLOduration=2.425568613 podStartE2EDuration="6.242396597s" totalImagesPullingTime="3.816827984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:28.241669842 +0000 UTC m=+246.021040425" watchObservedRunningTime="2026-09-18 19:42:28.242396597 +0000 UTC m=+246.021767180"
Sep 18 19:42:28.255474 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.255415    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv" podStartSLOduration=2.446517848 podStartE2EDuration="6.25539781s" totalImagesPullingTime="3.808879962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:28.254771753 +0000 UTC m=+246.034142336" watchObservedRunningTime="2026-09-18 19:42:28.25539781 +0000 UTC m=+246.034768397"
Sep 18 19:42:28.272921 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.272868    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd" podStartSLOduration=2.489736349 podStartE2EDuration="6.272845577s" totalImagesPullingTime="3.783109228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:28.272196042 +0000 UTC m=+246.051566625" watchObservedRunningTime="2026-09-18 19:42:28.272845577 +0000 UTC m=+246.052216160"
Sep 18 19:42:28.310660 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.310629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.311438 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.311416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.311514 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.311484    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-data-volume\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.311650 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.311635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-crio-socket\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.311704 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.311661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck29s\" (UniqueName: \"kubernetes.io/projected/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-api-access-ck29s\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412667    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-data-volume\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412744 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-crio-socket\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412950 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck29s\" (UniqueName: \"kubernetes.io/projected/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-api-access-ck29s\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412950 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412950 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-crio-socket\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.412950 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.412923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.413137 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.412994    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.413137 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.413013    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-data-volume\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.413137 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.412994    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.413137 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.413070    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:28.913049772 +0000 UTC m=+246.692420338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.413137 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.413093    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:28.913075475 +0000 UTC m=+246.692446036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.420828 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.420798    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck29s\" (UniqueName: \"kubernetes.io/projected/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-api-access-ck29s\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.916501 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.916456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.916675 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.916618    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.916675 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:28.916657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:28.916770 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.916684    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:29.916664591 +0000 UTC m=+247.696035151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.916770 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.916719    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.916770 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:28.916769    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:29.91675293 +0000 UTC m=+247.696123496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:29.008474 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.008438    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"]
Sep 18 19:42:29.012053 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.012029    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"
Sep 18 19:42:29.014136 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.014117    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:42:29.014572 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.014557    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fcc8w\""
Sep 18 19:42:29.014628 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.014590    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:42:29.020091 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.020071    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"]
Sep 18 19:42:29.118303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.118271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw2j\" (UniqueName: \"kubernetes.io/projected/af4c7473-7965-4e64-8c84-5e2ddbcf6a71-kube-api-access-xjw2j\") pod \"migrator-76794cbd6c-rmn48\" (UID: \"af4c7473-7965-4e64-8c84-5e2ddbcf6a71\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"
Sep 18 19:42:29.219851 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.219760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw2j\" (UniqueName: \"kubernetes.io/projected/af4c7473-7965-4e64-8c84-5e2ddbcf6a71-kube-api-access-xjw2j\") pod \"migrator-76794cbd6c-rmn48\" (UID: \"af4c7473-7965-4e64-8c84-5e2ddbcf6a71\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"
Sep 18 19:42:29.224470 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.224445    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/1.log"
Sep 18 19:42:29.224868 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.224850    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/0.log"
Sep 18 19:42:29.224980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.224895    2567 generic.go:356] "Generic (PLEG): container finished" podID="93f8637f-13ec-4134-ab92-45d02e00ce46" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d" exitCode=255
Sep 18 19:42:29.224980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.224970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" event={"ID":"93f8637f-13ec-4134-ab92-45d02e00ce46","Type":"ContainerDied","Data":"12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"}
Sep 18 19:42:29.225084 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.225021    2567 scope.go:117] "RemoveContainer" containerID="40ed196ff9c49325b5a295de41ea8bb43f026321b8dd8b4f15f18fc4189f418f"
Sep 18 19:42:29.225245 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.225219    2567 scope.go:117] "RemoveContainer" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"
Sep 18 19:42:29.225493 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:29.225446    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:42:29.227733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.227715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw2j\" (UniqueName: \"kubernetes.io/projected/af4c7473-7965-4e64-8c84-5e2ddbcf6a71-kube-api-access-xjw2j\") pod \"migrator-76794cbd6c-rmn48\" (UID: \"af4c7473-7965-4e64-8c84-5e2ddbcf6a71\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"
Sep 18 19:42:29.321602 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.321575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"
Sep 18 19:42:29.588310 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.588276    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48"]
Sep 18 19:42:29.591209 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:29.591185    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4c7473_7965_4e64_8c84_5e2ddbcf6a71.slice/crio-f7ed73d1c68cbda08537cacb59306aa484c39fe587a82ef4198a5b0b21eb806e WatchSource:0}: Error finding container f7ed73d1c68cbda08537cacb59306aa484c39fe587a82ef4198a5b0b21eb806e: Status 404 returned error can't find the container with id f7ed73d1c68cbda08537cacb59306aa484c39fe587a82ef4198a5b0b21eb806e
Sep 18 19:42:29.901849 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.901778    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-768dq_70ded459-00cc-4455-a8a4-a080de66c481/dns-node-resolver/0.log"
Sep 18 19:42:29.926760 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.926731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:29.926900 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:29.926795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:29.926900 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:29.926850    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:29.926967 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:29.926906    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:29.926967 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:29.926907    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:31.926891578 +0000 UTC m=+249.706262138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:29.927035 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:29.926967    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:31.926952278 +0000 UTC m=+249.706322845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:30.230907 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.230833    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/1.log"
Sep 18 19:42:30.231327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.231205    2567 scope.go:117] "RemoveContainer" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"
Sep 18 19:42:30.231427 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:30.231398    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:42:30.232422 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.232396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48" event={"ID":"af4c7473-7965-4e64-8c84-5e2ddbcf6a71","Type":"ContainerStarted","Data":"f7ed73d1c68cbda08537cacb59306aa484c39fe587a82ef4198a5b0b21eb806e"}
Sep 18 19:42:30.834975 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.834951    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:30.835092 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.835009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:30.835140 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:30.835087    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:42:30.835140 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:30.835109    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:30.835236 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:30.835150    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls podName:93d22068-2c02-49e6-b708-69d35f922a48 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:38.835132893 +0000 UTC m=+256.614503454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8q6vw" (UID: "93d22068-2c02-49e6-b708-69d35f922a48") : secret "samples-operator-tls" not found
Sep 18 19:42:30.835236 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:30.835164    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:38.835157585 +0000 UTC m=+256.614528145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:30.901290 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.901265    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6qv7_a369c464-bec9-42e5-87ef-63208b3d90f6/node-ca/0.log"
Sep 18 19:42:30.993565 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.993492    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-cm6z4"]
Sep 18 19:42:30.995417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.995403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:30.997259 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.997240    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 19:42:30.997259 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.997241    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 19:42:30.997453 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.997290    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 19:42:30.997840 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.997820    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 19:42:30.997840 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:30.997831    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-d7swd\""
Sep 18 19:42:31.004918 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.004893    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-cm6z4"]
Sep 18 19:42:31.137544 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.137512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-cabundle\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.137544 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.137547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-key\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.137747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.137577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zk7w\" (UniqueName: \"kubernetes.io/projected/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-kube-api-access-6zk7w\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.238118 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.238084    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-cabundle\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.238593 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.238126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-key\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.238593 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.238162    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zk7w\" (UniqueName: \"kubernetes.io/projected/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-kube-api-access-6zk7w\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.238941 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.238914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-cabundle\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.239592 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.239569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48" event={"ID":"af4c7473-7965-4e64-8c84-5e2ddbcf6a71","Type":"ContainerStarted","Data":"27cdc731a8e472658f585821d6278ad830262f5b9d6dd1c25e9f496b541c9274"}
Sep 18 19:42:31.239701 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.239654    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48" event={"ID":"af4c7473-7965-4e64-8c84-5e2ddbcf6a71","Type":"ContainerStarted","Data":"30d7af31e9aadea2b6a0d1fee5fa6b2ae3554fcdc6a8e9ed6a4aec9306c17169"}
Sep 18 19:42:31.240745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.240725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-signing-key\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.245911 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.245859    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zk7w\" (UniqueName: \"kubernetes.io/projected/88ba5a3a-cbd2-48e1-a86a-3b729570afaf-kube-api-access-6zk7w\") pod \"service-ca-55f6fc7b86-cm6z4\" (UID: \"88ba5a3a-cbd2-48e1-a86a-3b729570afaf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.254604 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.254565    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-rmn48" podStartSLOduration=2.110522814 podStartE2EDuration="3.254552533s" totalImagesPullingTime="1.144029719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:31.253600945 +0000 UTC m=+249.032971524" watchObservedRunningTime="2026-09-18 19:42:31.254552533 +0000 UTC m=+249.033923115"
Sep 18 19:42:31.303928 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.303898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4"
Sep 18 19:42:31.572832 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.572801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-cm6z4"]
Sep 18 19:42:31.575830 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:31.575801    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ba5a3a_cbd2_48e1_a86a_3b729570afaf.slice/crio-60018c84f5a359e1ae9f333b9929d645bbb9675b354af566fb353bbbeb277aa6 WatchSource:0}: Error finding container 60018c84f5a359e1ae9f333b9929d645bbb9675b354af566fb353bbbeb277aa6: Status 404 returned error can't find the container with id 60018c84f5a359e1ae9f333b9929d645bbb9675b354af566fb353bbbeb277aa6
Sep 18 19:42:31.902731 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.902659    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djj5j_11bdee4d-4727-49bd-91e4-8ba898621b9c/serve-healthcheck-canary/0.log"
Sep 18 19:42:31.944956 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.944916    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:31.945103 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:31.944998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:31.945103 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:31.945046    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:31.945230 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:31.945113    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:35.945097116 +0000 UTC m=+253.724467693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:31.945230 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:31.945152    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:31.945309 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:31.945238    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:35.945221623 +0000 UTC m=+253.724592189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:32.245580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:32.245494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4" event={"ID":"88ba5a3a-cbd2-48e1-a86a-3b729570afaf","Type":"ContainerStarted","Data":"07db776d448e61d746829920bc80549f0e54cee115b9c6d1774aa272036324e2"}
Sep 18 19:42:32.245580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:32.245543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4" event={"ID":"88ba5a3a-cbd2-48e1-a86a-3b729570afaf","Type":"ContainerStarted","Data":"60018c84f5a359e1ae9f333b9929d645bbb9675b354af566fb353bbbeb277aa6"}
Sep 18 19:42:32.260610 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:32.260564    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-cm6z4" podStartSLOduration=2.260549056 podStartE2EDuration="2.260549056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:32.260404982 +0000 UTC m=+250.039775566" watchObservedRunningTime="2026-09-18 19:42:32.260549056 +0000 UTC m=+250.039919616"
Sep 18 19:42:33.344289 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:33.344254    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:33.344688 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:33.344318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:33.344734 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:33.344707    2567 scope.go:117] "RemoveContainer" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"
Sep 18 19:42:33.344897 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:33.344876    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:42:35.980277 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:35.980230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:35.980782 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:35.980360    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:35.980782 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:35.980412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:35.980782 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:35.980434    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:43.980419376 +0000 UTC m=+261.759789937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:35.980782 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:35.980502    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:35.980782 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:35.980572    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls podName:10eec82f-5705-4e7f-845d-2ecbe2f09ff8 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:43.980559628 +0000 UTC m=+261.759930188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lth96" (UID: "10eec82f-5705-4e7f-845d-2ecbe2f09ff8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:36.182625 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:36.182586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:42:36.182799 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:36.182737    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:42:36.182799 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:36.182796    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs podName:695e01df-28bb-4aed-9e24-d53720c990fc nodeName:}" failed. No retries permitted until 2026-09-18 19:44:38.182779622 +0000 UTC m=+375.962150205 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs") pod "network-metrics-daemon-m6p9z" (UID: "695e01df-28bb-4aed-9e24-d53720c990fc") : secret "metrics-daemon-secret" not found
Sep 18 19:42:38.905354 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:38.905319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:38.905752 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:38.905404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:38.905752 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:38.905506    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:38.905752 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:38.905559    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls podName:a72572f6-da61-48e6-98cd-ab9d4478cd23 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:54.905545113 +0000 UTC m=+272.684915672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-nwq8p" (UID: "a72572f6-da61-48e6-98cd-ab9d4478cd23") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:42:38.907789 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:38.907768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93d22068-2c02-49e6-b708-69d35f922a48-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8q6vw\" (UID: \"93d22068-2c02-49e6-b708-69d35f922a48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:38.940745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:38.940717    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jqhtz\""
Sep 18 19:42:38.949414 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:38.949399    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"
Sep 18 19:42:39.215189 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:39.215166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw"]
Sep 18 19:42:39.269498 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:39.269472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw" event={"ID":"93d22068-2c02-49e6-b708-69d35f922a48","Type":"ContainerStarted","Data":"ca05459231f7fcc442082fe389374eb4de800040e8fafdbfc35c3213fdecc047"}
Sep 18 19:42:41.281683 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:41.281644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw" event={"ID":"93d22068-2c02-49e6-b708-69d35f922a48","Type":"ContainerStarted","Data":"f1d7e5d268b169e45cc39bf78e3566e341692e2a1c170a8a17503db51180a9ca"}
Sep 18 19:42:41.282032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:41.281739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw" event={"ID":"93d22068-2c02-49e6-b708-69d35f922a48","Type":"ContainerStarted","Data":"30287989496b10e53d1e941b1f4053c4b20ddd5bea943236d6165cb0bbd72192"}
Sep 18 19:42:41.296286 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:41.296238    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8q6vw" podStartSLOduration=17.765236804 podStartE2EDuration="19.296224562s" totalImagesPullingTime="1.530987758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:41.295775885 +0000 UTC m=+259.075146483" watchObservedRunningTime="2026-09-18 19:42:41.296224562 +0000 UTC m=+259.075595144"
Sep 18 19:42:44.046796 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.046755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:44.047297 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.046832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:44.047297 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.047286    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:44.049148 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.049122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10eec82f-5705-4e7f-845d-2ecbe2f09ff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lth96\" (UID: \"10eec82f-5705-4e7f-845d-2ecbe2f09ff8\") " pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:44.058124 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.058105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lth96"
Sep 18 19:42:44.325844 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.325769    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lth96"]
Sep 18 19:42:44.329218 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:44.329190    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10eec82f_5705_4e7f_845d_2ecbe2f09ff8.slice/crio-01cc138c23caddee2c91c5f4acdf0e4c94805f61c83044fd54fdf1c7d04e9ddc WatchSource:0}: Error finding container 01cc138c23caddee2c91c5f4acdf0e4c94805f61c83044fd54fdf1c7d04e9ddc: Status 404 returned error can't find the container with id 01cc138c23caddee2c91c5f4acdf0e4c94805f61c83044fd54fdf1c7d04e9ddc
Sep 18 19:42:44.349657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:44.349639    2567 scope.go:117] "RemoveContainer" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"
Sep 18 19:42:45.296815 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.296777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lth96" event={"ID":"10eec82f-5705-4e7f-845d-2ecbe2f09ff8","Type":"ContainerStarted","Data":"f924d91dd6275049c9ea0dbad0729471368f6f17ca0a877ae108c44d02e8b429"}
Sep 18 19:42:45.297253 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.296847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lth96" event={"ID":"10eec82f-5705-4e7f-845d-2ecbe2f09ff8","Type":"ContainerStarted","Data":"652a1c9d3c0e1c1cfd44884ef5fa4e0dc65ce52f4e4eec5430bbf983efaedf1d"}
Sep 18 19:42:45.297253 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.296873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lth96" event={"ID":"10eec82f-5705-4e7f-845d-2ecbe2f09ff8","Type":"ContainerStarted","Data":"01cc138c23caddee2c91c5f4acdf0e4c94805f61c83044fd54fdf1c7d04e9ddc"}
Sep 18 19:42:45.298384 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.298348    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:42:45.298771 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.298756    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/1.log"
Sep 18 19:42:45.298831 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.298787    2567 generic.go:356] "Generic (PLEG): container finished" podID="93f8637f-13ec-4134-ab92-45d02e00ce46" containerID="3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16" exitCode=255
Sep 18 19:42:45.298831 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.298810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" event={"ID":"93f8637f-13ec-4134-ab92-45d02e00ce46","Type":"ContainerDied","Data":"3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16"}
Sep 18 19:42:45.298899 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.298843    2567 scope.go:117] "RemoveContainer" containerID="12a6dd1400dba19e1433ee05ef99ae2f8e01115c844d62d597f69c1efa9def2d"
Sep 18 19:42:45.299144 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:45.299127    2567 scope.go:117] "RemoveContainer" containerID="3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16"
Sep 18 19:42:45.299353 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:45.299332    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:42:46.305438 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:46.305409    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:42:47.310506 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:47.310472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lth96" event={"ID":"10eec82f-5705-4e7f-845d-2ecbe2f09ff8","Type":"ContainerStarted","Data":"624cf3384cb4a5c15de11691453d02d37c882f726be6ec91c56ac2efef91fdde"}
Sep 18 19:42:47.326637 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:47.326583    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lth96" podStartSLOduration=17.37795216 podStartE2EDuration="19.32656892s" totalImagesPullingTime="1.94861676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:42:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:47.325785693 +0000 UTC m=+265.105156275" watchObservedRunningTime="2026-09-18 19:42:47.32656892 +0000 UTC m=+265.105939499"
Sep 18 19:42:51.407479 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:51.407450    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757975c5df-ltrdc"]
Sep 18 19:42:51.407821 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:51.407634    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-757975c5df-ltrdc" podUID="fca47ba9-859f-45b0-9285-0614e5101f21"
Sep 18 19:42:52.326311 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.326283    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:42:52.331513 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.331493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:42:52.412466 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.412819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412485    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.412819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412506    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.412819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412522    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nn4d\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.412819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412551    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.412819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412581    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.413072 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412892    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:42:52.413072 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412915    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:52.413072 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.412920    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:52.413236 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.413112    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets\") pod \"fca47ba9-859f-45b0-9285-0614e5101f21\" (UID: \"fca47ba9-859f-45b0-9285-0614e5101f21\") "
Sep 18 19:42:52.413405 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.413384    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fca47ba9-859f-45b0-9285-0614e5101f21-ca-trust-extracted\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.413473 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.413410    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-trusted-ca\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.413473 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.413424    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fca47ba9-859f-45b0-9285-0614e5101f21-registry-certificates\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.414960 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.414931    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:52.414960 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.414954    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:52.415126 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.415049    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d" (OuterVolumeSpecName: "kube-api-access-4nn4d") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "kube-api-access-4nn4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:52.415194 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.415179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fca47ba9-859f-45b0-9285-0614e5101f21" (UID: "fca47ba9-859f-45b0-9285-0614e5101f21"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:52.514298 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.514265    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-installation-pull-secrets\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.514298 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.514296    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fca47ba9-859f-45b0-9285-0614e5101f21-image-registry-private-configuration\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.514298 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.514306    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nn4d\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-kube-api-access-4nn4d\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:52.514494 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:52.514315    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-bound-sa-token\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:53.329120 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.329090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757975c5df-ltrdc"
Sep 18 19:42:53.344204 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.344180    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:53.344310 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.344248    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:42:53.344548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.344536    2567 scope.go:117] "RemoveContainer" containerID="3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16"
Sep 18 19:42:53.344695 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:42:53.344678    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:42:53.359572 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.359548    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757975c5df-ltrdc"]
Sep 18 19:42:53.364927 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.364901    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-757975c5df-ltrdc"]
Sep 18 19:42:53.420008 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:53.419979    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fca47ba9-859f-45b0-9285-0614e5101f21-registry-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:42:54.352209 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:54.352168    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca47ba9-859f-45b0-9285-0614e5101f21" path="/var/lib/kubelet/pods/fca47ba9-859f-45b0-9285-0614e5101f21/volumes"
Sep 18 19:42:54.929512 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:54.929474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:54.931898 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:54.931878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a72572f6-da61-48e6-98cd-ab9d4478cd23-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-nwq8p\" (UID: \"a72572f6-da61-48e6-98cd-ab9d4478cd23\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:55.130346 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:55.130313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-96pt7\""
Sep 18 19:42:55.138808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:55.138791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"
Sep 18 19:42:55.406463 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:55.406432    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p"]
Sep 18 19:42:55.409065 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:55.409033    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72572f6_da61_48e6_98cd_ab9d4478cd23.slice/crio-f07f4fe6aed90a8ca487f13ae395bcc39e4d11817b5988538553f2ae19fe7d9a WatchSource:0}: Error finding container f07f4fe6aed90a8ca487f13ae395bcc39e4d11817b5988538553f2ae19fe7d9a: Status 404 returned error can't find the container with id f07f4fe6aed90a8ca487f13ae395bcc39e4d11817b5988538553f2ae19fe7d9a
Sep 18 19:42:56.339962 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:56.339928    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p" event={"ID":"a72572f6-da61-48e6-98cd-ab9d4478cd23","Type":"ContainerStarted","Data":"f07f4fe6aed90a8ca487f13ae395bcc39e4d11817b5988538553f2ae19fe7d9a"}
Sep 18 19:42:57.346745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.346707    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p" event={"ID":"a72572f6-da61-48e6-98cd-ab9d4478cd23","Type":"ContainerStarted","Data":"4c6701b7434b3a303d4f208fd4db780a73013dd976dab5ec9c6812686917642f"}
Sep 18 19:42:57.367840 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.367789    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-nwq8p" podStartSLOduration=33.850290475 podStartE2EDuration="35.367773308s" totalImagesPullingTime="1.517482833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:57.366639814 +0000 UTC m=+275.146010398" watchObservedRunningTime="2026-09-18 19:42:57.367773308 +0000 UTC m=+275.147143890"
Sep 18 19:42:57.394384 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.394347    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"]
Sep 18 19:42:57.396529 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.396513    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:42:57.398383 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.398345    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 19:42:57.398487 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.398441    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zhm9c\""
Sep 18 19:42:57.404554 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.404536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"]
Sep 18 19:42:57.446890 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.446863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/730a5f3d-ee26-4529-8c97-34970f6dc091-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9bffz\" (UID: \"730a5f3d-ee26-4529-8c97-34970f6dc091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:42:57.547889 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.547861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/730a5f3d-ee26-4529-8c97-34970f6dc091-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9bffz\" (UID: \"730a5f3d-ee26-4529-8c97-34970f6dc091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:42:57.550234 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.550207    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/730a5f3d-ee26-4529-8c97-34970f6dc091-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9bffz\" (UID: \"730a5f3d-ee26-4529-8c97-34970f6dc091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:42:57.705642 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.705557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:42:57.971613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:57.971301    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"]
Sep 18 19:42:57.973822 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:42:57.973796    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730a5f3d_ee26_4529_8c97_34970f6dc091.slice/crio-8da49cef4b79c648217a4383cb1ba476078bf22319a8fe106fa62b94a80e83a2 WatchSource:0}: Error finding container 8da49cef4b79c648217a4383cb1ba476078bf22319a8fe106fa62b94a80e83a2: Status 404 returned error can't find the container with id 8da49cef4b79c648217a4383cb1ba476078bf22319a8fe106fa62b94a80e83a2
Sep 18 19:42:58.351848 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:42:58.351811    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz" event={"ID":"730a5f3d-ee26-4529-8c97-34970f6dc091","Type":"ContainerStarted","Data":"8da49cef4b79c648217a4383cb1ba476078bf22319a8fe106fa62b94a80e83a2"}
Sep 18 19:43:00.359001 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:00.358970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz" event={"ID":"730a5f3d-ee26-4529-8c97-34970f6dc091","Type":"ContainerStarted","Data":"9825b7381fc7e94f3113e1647ab2c4110c60cad81cd2bd05020f52356c8128c6"}
Sep 18 19:43:00.359383 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:00.359204    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:43:00.363560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:00.363539    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz"
Sep 18 19:43:00.373289 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:00.373252    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9bffz" podStartSLOduration=2.047408767 podStartE2EDuration="3.373238059s" totalImagesPullingTime="1.325829292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:00.371928305 +0000 UTC m=+278.151298887" watchObservedRunningTime="2026-09-18 19:43:00.373238059 +0000 UTC m=+278.152608687"
Sep 18 19:43:04.349983 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:04.349955    2567 scope.go:117] "RemoveContainer" containerID="3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16"
Sep 18 19:43:04.350472 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:43:04.350123    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-gkkdr_openshift-console-operator(93f8637f-13ec-4134-ab92-45d02e00ce46)\"" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podUID="93f8637f-13ec-4134-ab92-45d02e00ce46"
Sep 18 19:43:04.978438 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:43:04.978401    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7bsqr" podUID="06f430df-4b07-4547-b71e-68541a5134ab"
Sep 18 19:43:05.373596 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.373565    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:05.819936 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.819903    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7mkpp"]
Sep 18 19:43:05.823480 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.823453    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.824032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.823990    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"]
Sep 18 19:43:05.825327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.825295    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:43:05.825825 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.825804    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:43:05.825981 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.825961    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:43:05.826166 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.826142    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:43:05.826243 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.826015    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jkr4l\""
Sep 18 19:43:05.826566 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.826548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.829089 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.829067    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 19:43:05.829191 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.829093    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w7px4\""
Sep 18 19:43:05.829191 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.829154    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 19:43:05.829316 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.829281    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:43:05.841961 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.841940    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"]
Sep 18 19:43:05.906653 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-root\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.906830 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906664    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.906830 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.906830 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-metrics-client-ca\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.906969 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906840    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.906969 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.906969 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906886    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-wtmp\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.906969 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-textfile\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.906987    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.907015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f1ba49fe-4fb4-47ed-a640-7325e7f89518-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.907042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wvg\" (UniqueName: \"kubernetes.io/projected/ae0aa986-cb47-49b5-a290-022d3637fb78-kube-api-access-s5wvg\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.907064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.907131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.907100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q44b\" (UniqueName: \"kubernetes.io/projected/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-api-access-5q44b\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:05.907348 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:05.907138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-sys\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.007806 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-wtmp\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-textfile\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f1ba49fe-4fb4-47ed-a640-7325e7f89518-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wvg\" (UniqueName: \"kubernetes.io/projected/ae0aa986-cb47-49b5-a290-022d3637fb78-kube-api-access-s5wvg\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.007985    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-wtmp\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008193    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-textfile\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008524 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f1ba49fe-4fb4-47ed-a640-7325e7f89518-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008674 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q44b\" (UniqueName: \"kubernetes.io/projected/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-api-access-5q44b\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008786 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-sys\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008786 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008722    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-root\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008786 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.008938 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008938 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-metrics-client-ca\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008938 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-sys\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.008938 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008912    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.009135 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.008952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.009328 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.009310    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-metrics-client-ca\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.009539 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:43:06.009523    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 19:43:06.009682 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:43:06.009673    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls podName:ae0aa986-cb47-49b5-a290-022d3637fb78 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:06.509654354 +0000 UTC m=+284.289024929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls") pod "node-exporter-7mkpp" (UID: "ae0aa986-cb47-49b5-a290-022d3637fb78") : secret "node-exporter-tls" not found
Sep 18 19:43:06.009770 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.009748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae0aa986-cb47-49b5-a290-022d3637fb78-root\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.009829 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.009703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.010011 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.009988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba49fe-4fb4-47ed-a640-7325e7f89518-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.010688 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.010449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.010774 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.010753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.011353 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.011330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.011526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.011510    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.015759 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.015734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wvg\" (UniqueName: \"kubernetes.io/projected/ae0aa986-cb47-49b5-a290-022d3637fb78-kube-api-access-s5wvg\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.015847 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.015740    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q44b\" (UniqueName: \"kubernetes.io/projected/f1ba49fe-4fb4-47ed-a640-7325e7f89518-kube-api-access-5q44b\") pod \"kube-state-metrics-84c885658c-gwxfz\" (UID: \"f1ba49fe-4fb4-47ed-a640-7325e7f89518\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.140439 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.140338    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"
Sep 18 19:43:06.410573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.410531    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-gwxfz"]
Sep 18 19:43:06.412899 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:06.412867    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ba49fe_4fb4_47ed_a640_7325e7f89518.slice/crio-d3e982e9e04fbf70df3c35b97b16b58a99c8dd06c68a24c0f6dfd99b98422b55 WatchSource:0}: Error finding container d3e982e9e04fbf70df3c35b97b16b58a99c8dd06c68a24c0f6dfd99b98422b55: Status 404 returned error can't find the container with id d3e982e9e04fbf70df3c35b97b16b58a99c8dd06c68a24c0f6dfd99b98422b55
Sep 18 19:43:06.512900 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.512866    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.515110 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.515092    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae0aa986-cb47-49b5-a290-022d3637fb78-node-exporter-tls\") pod \"node-exporter-7mkpp\" (UID: \"ae0aa986-cb47-49b5-a290-022d3637fb78\") " pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.735622 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:06.735543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mkpp"
Sep 18 19:43:06.745684 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:06.745653    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0aa986_cb47_49b5_a290_022d3637fb78.slice/crio-83f44afc17bc8ac9521ed5821f9256c00213a7bf804c0b329d6b79b5afdf5136 WatchSource:0}: Error finding container 83f44afc17bc8ac9521ed5821f9256c00213a7bf804c0b329d6b79b5afdf5136: Status 404 returned error can't find the container with id 83f44afc17bc8ac9521ed5821f9256c00213a7bf804c0b329d6b79b5afdf5136
Sep 18 19:43:07.383179 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.383137    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mkpp" event={"ID":"ae0aa986-cb47-49b5-a290-022d3637fb78","Type":"ContainerStarted","Data":"83f44afc17bc8ac9521ed5821f9256c00213a7bf804c0b329d6b79b5afdf5136"}
Sep 18 19:43:07.384652 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.384625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz" event={"ID":"f1ba49fe-4fb4-47ed-a640-7325e7f89518","Type":"ContainerStarted","Data":"d3e982e9e04fbf70df3c35b97b16b58a99c8dd06c68a24c0f6dfd99b98422b55"}
Sep 18 19:43:07.826684 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.826648    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"]
Sep 18 19:43:07.830044 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.830019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.832015 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.831988    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:43:07.833054 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833031    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:43:07.833163 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833092    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:43:07.833442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833357    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:43:07.833442 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833402    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qxnl8\""
Sep 18 19:43:07.833589 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833467    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:43:07.833655 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.833639    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cjn677cdgahtr\""
Sep 18 19:43:07.842960 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.842937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"]
Sep 18 19:43:07.929860 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.929822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930007 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.929879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930007 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.929944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.930032    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-grpc-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.930112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930199 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.930151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-metrics-client-ca\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.930203    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md69t\" (UniqueName: \"kubernetes.io/projected/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-kube-api-access-md69t\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:07.930281 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:07.930244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031328 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-grpc-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031462 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-metrics-client-ca\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031621 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md69t\" (UniqueName: \"kubernetes.io/projected/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-kube-api-access-md69t\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031621 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031621 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.031621 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.031601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.032579 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.032554    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-metrics-client-ca\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.034411 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.034386    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.034510 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.034495    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.034849 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.034824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-grpc-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.034971 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.034948    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.035203 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.035183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.035334 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.035313    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-secret-thanos-querier-tls\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.042862 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.042840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md69t\" (UniqueName: \"kubernetes.io/projected/f6aeb7ed-56c2-4533-94d0-aa4b38d01961-kube-api-access-md69t\") pod \"thanos-querier-84fb5f84b9-gnpxd\" (UID: \"f6aeb7ed-56c2-4533-94d0-aa4b38d01961\") " pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.142150 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.142127    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:08.335318 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.335279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:08.337804 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.337743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f430df-4b07-4547-b71e-68541a5134ab-metrics-tls\") pod \"dns-default-7bsqr\" (UID: \"06f430df-4b07-4547-b71e-68541a5134ab\") " pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:08.376047 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.376018    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j9gtw\""
Sep 18 19:43:08.384322 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.384298    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:08.391242 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.391214    2567 generic.go:356] "Generic (PLEG): container finished" podID="ae0aa986-cb47-49b5-a290-022d3637fb78" containerID="a5fee810c61ec0b4d781f3debb3375d21fc051f5e480e63c834b2a63064a260b" exitCode=0
Sep 18 19:43:08.391386 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.391305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mkpp" event={"ID":"ae0aa986-cb47-49b5-a290-022d3637fb78","Type":"ContainerDied","Data":"a5fee810c61ec0b4d781f3debb3375d21fc051f5e480e63c834b2a63064a260b"}
Sep 18 19:43:08.393685 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.393642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz" event={"ID":"f1ba49fe-4fb4-47ed-a640-7325e7f89518","Type":"ContainerStarted","Data":"318a7223ca373a93ffa93895207fd4b7117704373e5425a49464edf6ddb651d6"}
Sep 18 19:43:08.393809 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.393716    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz" event={"ID":"f1ba49fe-4fb4-47ed-a640-7325e7f89518","Type":"ContainerStarted","Data":"3f2a992074d5fa0b80f23d00324826250ef6a3ce00bd31b4309b86311b0c7d0f"}
Sep 18 19:43:08.393809 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.393754    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz" event={"ID":"f1ba49fe-4fb4-47ed-a640-7325e7f89518","Type":"ContainerStarted","Data":"79434b4dd9b28dc2254bcd3438f837737bad0f30b6b9af5859b6ce45aaa3d915"}
Sep 18 19:43:08.428530 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.428433    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"]
Sep 18 19:43:08.429157 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:08.429095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aeb7ed_56c2_4533_94d0_aa4b38d01961.slice/crio-32e8a8ae71224d20bae0e66914019da821531578fcb03c61a75b7a60828852df WatchSource:0}: Error finding container 32e8a8ae71224d20bae0e66914019da821531578fcb03c61a75b7a60828852df: Status 404 returned error can't find the container with id 32e8a8ae71224d20bae0e66914019da821531578fcb03c61a75b7a60828852df
Sep 18 19:43:08.442803 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.442243    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-gwxfz" podStartSLOduration=1.8821485999999998 podStartE2EDuration="3.442221435s" totalImagesPullingTime="1.560072835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:08.440465676 +0000 UTC m=+286.219836260" watchObservedRunningTime="2026-09-18 19:43:08.442221435 +0000 UTC m=+286.221592018"
Sep 18 19:43:08.666301 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:08.666258    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7bsqr"]
Sep 18 19:43:08.668820 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:08.668794    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f430df_4b07_4547_b71e_68541a5134ab.slice/crio-6d8bd4287e1d735c4ce16bc9bdaae15e707c33130f5139c3499074ade66cfe4e WatchSource:0}: Error finding container 6d8bd4287e1d735c4ce16bc9bdaae15e707c33130f5139c3499074ade66cfe4e: Status 404 returned error can't find the container with id 6d8bd4287e1d735c4ce16bc9bdaae15e707c33130f5139c3499074ade66cfe4e
Sep 18 19:43:09.402794 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:09.402502    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mkpp" event={"ID":"ae0aa986-cb47-49b5-a290-022d3637fb78","Type":"ContainerStarted","Data":"4aabcbf50d0a0e16960d0ac7c5732ff5b372ad54edcbf6786fc10b4dced9742e"}
Sep 18 19:43:09.402794 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:09.402561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mkpp" event={"ID":"ae0aa986-cb47-49b5-a290-022d3637fb78","Type":"ContainerStarted","Data":"9ad1a2a5e7d696066fda6c593e10ec9f305b3a2e66d56a263267d701deeb534d"}
Sep 18 19:43:09.404516 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:09.404430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"32e8a8ae71224d20bae0e66914019da821531578fcb03c61a75b7a60828852df"}
Sep 18 19:43:09.407205 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:09.407171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7bsqr" event={"ID":"06f430df-4b07-4547-b71e-68541a5134ab","Type":"ContainerStarted","Data":"6d8bd4287e1d735c4ce16bc9bdaae15e707c33130f5139c3499074ade66cfe4e"}
Sep 18 19:43:09.423015 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:09.422965    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7mkpp" podStartSLOduration=3.111506996 podStartE2EDuration="4.422937703s" totalImagesPullingTime="1.229253796s" totalInitContainerRuntime="82.176911ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:09.422178757 +0000 UTC m=+287.201549341" watchObservedRunningTime="2026-09-18 19:43:09.422937703 +0000 UTC m=+287.202308288"
Sep 18 19:43:10.129615 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.129581    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b55d657d-pmjh5"]
Sep 18 19:43:10.132141 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.132123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.134690 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.134669    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 19:43:10.134812 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.134739    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jvvrb\""
Sep 18 19:43:10.134812 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.134778    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:43:10.134925 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.134887    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 19:43:10.134976 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.134961    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a03i2vpdvdl8m\""
Sep 18 19:43:10.135282 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.135265    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 19:43:10.142255 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.141827    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b55d657d-pmjh5"]
Sep 18 19:43:10.253656 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-client-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.253820 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253666    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-audit-log\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.253820 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92bf\" (UniqueName: \"kubernetes.io/projected/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-kube-api-access-p92bf\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.253820 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-tls\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.253956 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-client-certs\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.253956 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.254047 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.253978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-metrics-server-audit-profiles\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.354895 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.354859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-client-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355020 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.354902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-audit-log\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355123 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92bf\" (UniqueName: \"kubernetes.io/projected/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-kube-api-access-p92bf\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355140    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-tls\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-client-certs\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355437 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-audit-log\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355437 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-metrics-server-audit-profiles\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.355964 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.355940    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.356446 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.356410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-metrics-server-audit-profiles\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.357682 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.357655    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-tls\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.357838 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.357820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-client-ca-bundle\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.357894 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.357867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-secret-metrics-server-client-certs\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.362886 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.362864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92bf\" (UniqueName: \"kubernetes.io/projected/f5bc57e6-387c-4b5b-9c42-57b4fbd9564d-kube-api-access-p92bf\") pod \"metrics-server-7b55d657d-pmjh5\" (UID: \"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d\") " pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:10.445542 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:10.445465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:11.156913 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.156829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b55d657d-pmjh5"]
Sep 18 19:43:11.161208 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:11.161175    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bc57e6_387c_4b5b_9c42_57b4fbd9564d.slice/crio-a20b18b84e8662ab8c9b9577673efd64390e2c5809608b5e9920985bb47fd31e WatchSource:0}: Error finding container a20b18b84e8662ab8c9b9577673efd64390e2c5809608b5e9920985bb47fd31e: Status 404 returned error can't find the container with id a20b18b84e8662ab8c9b9577673efd64390e2c5809608b5e9920985bb47fd31e
Sep 18 19:43:11.416564 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.416479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"8fe23442721af95d466ac492d545bfb5975f8e8c04729da0c65768968c470359"}
Sep 18 19:43:11.416564 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.416546    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"9f5a8f88ec0dd48f48ad6e020acf117bb793b239ef7c51083119140df0a2c08c"}
Sep 18 19:43:11.416764 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.416582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"8daf3b104634ce388aaf07efca9c26eb3551d84c8ead9920019635b8ea796fd2"}
Sep 18 19:43:11.417530 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.417511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5" event={"ID":"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d","Type":"ContainerStarted","Data":"a20b18b84e8662ab8c9b9577673efd64390e2c5809608b5e9920985bb47fd31e"}
Sep 18 19:43:11.419145 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.419127    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7bsqr" event={"ID":"06f430df-4b07-4547-b71e-68541a5134ab","Type":"ContainerStarted","Data":"ca999cfcddc94d419624b0b08b8a53523a2d0461f48edcd973dea52fc9caf303"}
Sep 18 19:43:11.419229 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.419160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7bsqr" event={"ID":"06f430df-4b07-4547-b71e-68541a5134ab","Type":"ContainerStarted","Data":"5e045bde072f85722933c32195278b3878cf79f245dd6073c636cd259037bd61"}
Sep 18 19:43:11.419286 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.419270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:11.434733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:11.434692    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7bsqr" podStartSLOduration=251.217870491 podStartE2EDuration="4m13.434676619s" totalImagesPullingTime="2.216806128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:11.434441388 +0000 UTC m=+289.213811963" watchObservedRunningTime="2026-09-18 19:43:11.434676619 +0000 UTC m=+289.214047203"
Sep 18 19:43:12.150436 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.150396    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:43:12.154765 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.154748    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.162552 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.162156    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:43:12.162552 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.162244    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:43:12.162552 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.162315    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cnnd0p2jiu0e9\""
Sep 18 19:43:12.162831 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.162699    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:43:12.163323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.163031    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:43:12.163323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.163054    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:43:12.163323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.163089    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:43:12.163323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.163248    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:43:12.164417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.164158    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:43:12.164417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.164261    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:43:12.164417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.164289    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:43:12.164417 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.164312    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:43:12.166458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.165686    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:43:12.166458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.165728    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7hhm6\""
Sep 18 19:43:12.166458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.166299    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:43:12.170598 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.170150    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:43:12.172007 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.171989    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:43:12.271278 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271278 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271278 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271443    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271496    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271681    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.271769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.272017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.272017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.272017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271842    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.272017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271857    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.272017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.271871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72l7\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373027 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.372996    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373104    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72l7\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373396    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373479    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.373543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.374016 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.373627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.374822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.375735    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.376243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.376441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.376439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.377006 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.376982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.377810 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.377707    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.378286 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.378000    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.378286 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.378233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.378924 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.378477    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.378924 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.378880    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.379220 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.379175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.379220 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.379108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.380074 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.380000    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.380403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.380349    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.380601 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.380580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.380908 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.380884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.381358 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.381341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72l7\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7\") pod \"prometheus-k8s-0\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.427891 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.427860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"36f213824378073b35fec4ea30183cd4035229f466b61c365acfb8cd4c03e06e"}
Sep 18 19:43:12.428020 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.427938    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"3153d8024bfc4bac67c92632b6feb59681f7123bcc994ad9b364345b32b26d1e"}
Sep 18 19:43:12.428020 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.427977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" event={"ID":"f6aeb7ed-56c2-4533-94d0-aa4b38d01961","Type":"ContainerStarted","Data":"0894c3bf18a84858e9cbbc5bd8b9d0332a42371fc61341e0ea8ad399db294293"}
Sep 18 19:43:12.452902 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.452819    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd" podStartSLOduration=2.082300359 podStartE2EDuration="5.452800715s" totalImagesPullingTime="3.370500356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:12.451818583 +0000 UTC m=+290.231189165" watchObservedRunningTime="2026-09-18 19:43:12.452800715 +0000 UTC m=+290.232171297"
Sep 18 19:43:12.474032 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.473992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:12.926565 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:12.926486    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:43:12.929649 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:43:12.929621    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dd894c_5d68_4f63_81b4_ec0ddc830acc.slice/crio-3e706a463b9ba2738443ce88b910ef9b1cbf260b88022aa775fc0fd06c4a5344 WatchSource:0}: Error finding container 3e706a463b9ba2738443ce88b910ef9b1cbf260b88022aa775fc0fd06c4a5344: Status 404 returned error can't find the container with id 3e706a463b9ba2738443ce88b910ef9b1cbf260b88022aa775fc0fd06c4a5344
Sep 18 19:43:13.435165 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:13.435117    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5" event={"ID":"f5bc57e6-387c-4b5b-9c42-57b4fbd9564d","Type":"ContainerStarted","Data":"90aa4e75a649767e355e2fe108b38265afb4eedea306d2def8025206cc067a49"}
Sep 18 19:43:13.436828 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:13.436800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"3e706a463b9ba2738443ce88b910ef9b1cbf260b88022aa775fc0fd06c4a5344"}
Sep 18 19:43:13.437214 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:13.437197    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:13.454424 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:13.454377    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5" podStartSLOduration=1.940279829 podStartE2EDuration="3.454350173s" totalImagesPullingTime="1.514070344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:13.452960956 +0000 UTC m=+291.232331542" watchObservedRunningTime="2026-09-18 19:43:13.454350173 +0000 UTC m=+291.233720756"
Sep 18 19:43:14.441609 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:14.441575    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95" exitCode=0
Sep 18 19:43:14.442041 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:14.441658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95"}
Sep 18 19:43:15.349487 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:43:15.349435    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-m6p9z" podUID="695e01df-28bb-4aed-9e24-d53720c990fc"
Sep 18 19:43:18.462188 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"}
Sep 18 19:43:18.462558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462215    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"}
Sep 18 19:43:18.462558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"}
Sep 18 19:43:18.462558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"}
Sep 18 19:43:18.462558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"}
Sep 18 19:43:18.462558 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.462315    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerStarted","Data":"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"}
Sep 18 19:43:18.491238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:18.491188    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.39840631 podStartE2EDuration="6.49117248s" totalImagesPullingTime="4.073830761s" totalInitContainerRuntime="18.935409ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:18.489259442 +0000 UTC m=+296.268630023" watchObservedRunningTime="2026-09-18 19:43:18.49117248 +0000 UTC m=+296.270543061"
Sep 18 19:43:19.348534 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:19.348507    2567 scope.go:117] "RemoveContainer" containerID="3561367750b3a4ebfdcfe6e21ef8e7b2f1d471d70b18a36be31acad29680fb16"
Sep 18 19:43:19.449648 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:19.449623    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84fb5f84b9-gnpxd"
Sep 18 19:43:19.467730 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:19.467710    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:43:19.468033 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:19.467822    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" event={"ID":"93f8637f-13ec-4134-ab92-45d02e00ce46","Type":"ContainerStarted","Data":"13304cb52a6e7a8985e37181819552b631a34f83f84c4c69c9e5e5f1d3007da2"}
Sep 18 19:43:19.487148 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:19.487101    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr" podStartSLOduration=53.777178421 podStartE2EDuration="57.487082152s" totalImagesPullingTime="3.709903731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:19.485999909 +0000 UTC m=+297.265370492" watchObservedRunningTime="2026-09-18 19:43:19.487082152 +0000 UTC m=+297.266452735"
Sep 18 19:43:21.430190 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:21.430155    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7bsqr"
Sep 18 19:43:22.474347 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:22.474321    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:24.210042 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:24.210016    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:43:24.211605 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:24.211580    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:43:24.221820 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:24.221804    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:27.347501 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:27.347472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:43:29.468943 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:29.468906    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:43:29.472726 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:29.472707    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-gkkdr"
Sep 18 19:43:30.446094 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:30.446067    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:30.446248 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:30.446108    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:43.569458 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:43.569425    2567 generic.go:356] "Generic (PLEG): container finished" podID="6cd8f8e0-03ee-4342-b072-5d602b5a60be" containerID="eaf04bcda04e6e694d2d2c25ec977728e6a1c9a8837e0fa3bc1b4da9b6836b7f" exitCode=0
Sep 18 19:43:43.569916 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:43.569501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5" event={"ID":"6cd8f8e0-03ee-4342-b072-5d602b5a60be","Type":"ContainerDied","Data":"eaf04bcda04e6e694d2d2c25ec977728e6a1c9a8837e0fa3bc1b4da9b6836b7f"}
Sep 18 19:43:43.569916 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:43.569857    2567 scope.go:117] "RemoveContainer" containerID="eaf04bcda04e6e694d2d2c25ec977728e6a1c9a8837e0fa3bc1b4da9b6836b7f"
Sep 18 19:43:44.574517 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:44.574485    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-zl9p5" event={"ID":"6cd8f8e0-03ee-4342-b072-5d602b5a60be","Type":"ContainerStarted","Data":"a65181aefcfe5b42f2502ed04aa2abe40d7485f9a62448f86ae075d03e11a8e8"}
Sep 18 19:43:50.451657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:50.451626    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:50.455460 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:50.455437    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b55d657d-pmjh5"
Sep 18 19:43:53.610798 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:53.610768    2567 generic.go:356] "Generic (PLEG): container finished" podID="093c8031-69fd-4cde-84f0-cdd7614bdb8b" containerID="ebee031f8723baed5715b4bfe9741788b555780f44dbd3f53e99d5c366d23303" exitCode=0
Sep 18 19:43:53.611179 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:53.610823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd" event={"ID":"093c8031-69fd-4cde-84f0-cdd7614bdb8b","Type":"ContainerDied","Data":"ebee031f8723baed5715b4bfe9741788b555780f44dbd3f53e99d5c366d23303"}
Sep 18 19:43:53.611179 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:53.611134    2567 scope.go:117] "RemoveContainer" containerID="ebee031f8723baed5715b4bfe9741788b555780f44dbd3f53e99d5c366d23303"
Sep 18 19:43:54.617093 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:54.617063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4vsvd" event={"ID":"093c8031-69fd-4cde-84f0-cdd7614bdb8b","Type":"ContainerStarted","Data":"1797741a9bfd17f4016f897d6b1660864bdb7cccb8ebfeb9dfad17739be76a44"}
Sep 18 19:43:58.634834 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:58.634804    2567 generic.go:356] "Generic (PLEG): container finished" podID="051ad5ae-9e24-48fa-940d-fe0ea43a9491" containerID="78e514a0a562b83c97896236e19473ed74d164814bbc573829e6bb2b8f418f58" exitCode=0
Sep 18 19:43:58.634834 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:58.634839    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv" event={"ID":"051ad5ae-9e24-48fa-940d-fe0ea43a9491","Type":"ContainerDied","Data":"78e514a0a562b83c97896236e19473ed74d164814bbc573829e6bb2b8f418f58"}
Sep 18 19:43:58.635279 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:58.635097    2567 scope.go:117] "RemoveContainer" containerID="78e514a0a562b83c97896236e19473ed74d164814bbc573829e6bb2b8f418f58"
Sep 18 19:43:59.644303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:43:59.644265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-csqxv" event={"ID":"051ad5ae-9e24-48fa-940d-fe0ea43a9491","Type":"ContainerStarted","Data":"2b39d33bd1ad29115531a0951813a4adcfda70eec9ed7cd59cacc7d381a79915"}
Sep 18 19:44:12.474234 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:12.474150    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:12.492902 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:12.492879    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:12.717652 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:12.717621    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:30.480385 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480337    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480658    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="prometheus" containerID="cri-o://6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221" gracePeriod=600
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480691    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="thanos-sidecar" containerID="cri-o://99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9" gracePeriod=600
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480712    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805" gracePeriod=600
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480724    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="config-reloader" containerID="cri-o://6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e" gracePeriod=600
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480659    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy" containerID="cri-o://f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b" gracePeriod=600
Sep 18 19:44:30.480892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.480852    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-web" containerID="cri-o://d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7" gracePeriod=600
Sep 18 19:44:30.780121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780092    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805" exitCode=0
Sep 18 19:44:30.780121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780114    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b" exitCode=0
Sep 18 19:44:30.780121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780120    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9" exitCode=0
Sep 18 19:44:30.780121 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780127    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e" exitCode=0
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780132    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221" exitCode=0
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"}
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"}
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780254    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"}
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"}
Sep 18 19:44:30.780387 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:30.780310    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"}
Sep 18 19:44:31.725185 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.725163    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:31.788241 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.788206    2567 generic.go:356] "Generic (PLEG): container finished" podID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerID="d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7" exitCode=0
Sep 18 19:44:31.788416 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.788261    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"}
Sep 18 19:44:31.788416 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.788310    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:31.788416 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.788400    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46dd894c-5d68-4f63-81b4-ec0ddc830acc","Type":"ContainerDied","Data":"3e706a463b9ba2738443ce88b910ef9b1cbf260b88022aa775fc0fd06c4a5344"}
Sep 18 19:44:31.788532 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.788446    2567 scope.go:117] "RemoveContainer" containerID="851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"
Sep 18 19:44:31.796660 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.796640    2567 scope.go:117] "RemoveContainer" containerID="f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"
Sep 18 19:44:31.804841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.804824    2567 scope.go:117] "RemoveContainer" containerID="d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"
Sep 18 19:44:31.811413 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.811395    2567 scope.go:117] "RemoveContainer" containerID="99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"
Sep 18 19:44:31.814536 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814518    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814622 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814622 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814574    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814622 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814597    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814623    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814660    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814704    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814727    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814751    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.814790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814783    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814817    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814841    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w72l7\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814899    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814933    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814965    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.814994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815027    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815042    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.815073 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815064    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815089    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls\") pod \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\" (UID: \"46dd894c-5d68-4f63-81b4-ec0ddc830acc\") "
Sep 18 19:44:31.815537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.815482    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.816380 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.816034    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.816380 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.816057    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.816380 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.816311    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.816745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.816724    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.817043 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.817021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.817126 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.817108    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:31.817252 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.817226    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.819192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.818798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:31.820745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.820721    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out" (OuterVolumeSpecName: "config-out") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:31.820745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.820738    2567 scope.go:117] "RemoveContainer" containerID="6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"
Sep 18 19:44:31.820890 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.820736    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.820890 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.820803    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.820890 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.820765    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config" (OuterVolumeSpecName: "config") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.821747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.821715    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7" (OuterVolumeSpecName: "kube-api-access-w72l7") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "kube-api-access-w72l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:31.821833 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.821805    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.822222 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.822199    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.822303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.822223    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.822303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.822268    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.822631 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.822601    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:31.834476 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.834452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config" (OuterVolumeSpecName: "web-config") pod "46dd894c-5d68-4f63-81b4-ec0ddc830acc" (UID: "46dd894c-5d68-4f63-81b4-ec0ddc830acc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:31.839401 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.839383    2567 scope.go:117] "RemoveContainer" containerID="6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"
Sep 18 19:44:31.846516 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.846486    2567 scope.go:117] "RemoveContainer" containerID="bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95"
Sep 18 19:44:31.852661 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.852646    2567 scope.go:117] "RemoveContainer" containerID="851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"
Sep 18 19:44:31.852943 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.852926    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805\": container with ID starting with 851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805 not found: ID does not exist" containerID="851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"
Sep 18 19:44:31.852986 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.852950    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805"} err="failed to get container status \"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805\": rpc error: code = NotFound desc = could not find container \"851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805\": container with ID starting with 851ce5bdb3220e4b1fb006ebc4c6add5db881c7f0c2344278006091336d81805 not found: ID does not exist"
Sep 18 19:44:31.852986 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.852978    2567 scope.go:117] "RemoveContainer" containerID="f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"
Sep 18 19:44:31.853199 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.853184    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b\": container with ID starting with f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b not found: ID does not exist" containerID="f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"
Sep 18 19:44:31.853239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853204    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b"} err="failed to get container status \"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b\": rpc error: code = NotFound desc = could not find container \"f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b\": container with ID starting with f8180a24c9415829ff5b88d16d7324853b9e978eb530b7b1667cbcd2ed2ee89b not found: ID does not exist"
Sep 18 19:44:31.853239 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853218    2567 scope.go:117] "RemoveContainer" containerID="d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"
Sep 18 19:44:31.853480 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.853462    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7\": container with ID starting with d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7 not found: ID does not exist" containerID="d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"
Sep 18 19:44:31.853540 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853483    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7"} err="failed to get container status \"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7\": rpc error: code = NotFound desc = could not find container \"d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7\": container with ID starting with d317040dddc1a31be7257dbefaa6d7eb3c48735972242c39c7e6f0df279c64d7 not found: ID does not exist"
Sep 18 19:44:31.853540 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853496    2567 scope.go:117] "RemoveContainer" containerID="99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"
Sep 18 19:44:31.853761 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.853743    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9\": container with ID starting with 99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9 not found: ID does not exist" containerID="99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"
Sep 18 19:44:31.853812 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853768    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9"} err="failed to get container status \"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9\": rpc error: code = NotFound desc = could not find container \"99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9\": container with ID starting with 99fef16af4a90bd1dd0a49494edab6703c16b1b1a3d95236f0fbc46b7e81cfe9 not found: ID does not exist"
Sep 18 19:44:31.853812 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.853783    2567 scope.go:117] "RemoveContainer" containerID="6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"
Sep 18 19:44:31.854021 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.854004    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e\": container with ID starting with 6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e not found: ID does not exist" containerID="6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"
Sep 18 19:44:31.854061 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.854025    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e"} err="failed to get container status \"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e\": rpc error: code = NotFound desc = could not find container \"6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e\": container with ID starting with 6725037ae27c62e36945790d8288844d5b243dda7b2d262c83b143b724f93d8e not found: ID does not exist"
Sep 18 19:44:31.854061 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.854038    2567 scope.go:117] "RemoveContainer" containerID="6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"
Sep 18 19:44:31.854282 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.854264    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221\": container with ID starting with 6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221 not found: ID does not exist" containerID="6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"
Sep 18 19:44:31.854350 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.854298    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221"} err="failed to get container status \"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221\": rpc error: code = NotFound desc = could not find container \"6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221\": container with ID starting with 6c4ff444a9e57428d636140b2fabb49cf1a531b4c1caaa458b49b29b0686f221 not found: ID does not exist"
Sep 18 19:44:31.854350 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.854322    2567 scope.go:117] "RemoveContainer" containerID="bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95"
Sep 18 19:44:31.854579 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:44:31.854562    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95\": container with ID starting with bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95 not found: ID does not exist" containerID="bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95"
Sep 18 19:44:31.854615 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.854583    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95"} err="failed to get container status \"bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95\": rpc error: code = NotFound desc = could not find container \"bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95\": container with ID starting with bd580af8366d6a9a25497ede91d0af812da9541c161c4162321468e7fe7a2a95 not found: ID does not exist"
Sep 18 19:44:31.916104 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916083    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916104 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916101    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916111    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916120    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config-out\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916130    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-grpc-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916137    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-web-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916145    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916153    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916160    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-db\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916169    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916177    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-kube-rbac-proxy\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916187    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72l7\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-kube-api-access-w72l7\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916194    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916203    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916211 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916211    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-secret-metrics-client-certs\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916602 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916220    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-metrics-client-ca\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916602 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916228    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46dd894c-5d68-4f63-81b4-ec0ddc830acc-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916602 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916237    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46dd894c-5d68-4f63-81b4-ec0ddc830acc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:31.916602 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:31.916245    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46dd894c-5d68-4f63-81b4-ec0ddc830acc-tls-assets\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:44:32.110972 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.110947    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:32.114769 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.114748    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:32.137903 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.137881    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:32.138178 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138167    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="prometheus"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138179    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="prometheus"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138194    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="config-reloader"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138199    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="config-reloader"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138207    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138212    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy"
Sep 18 19:44:32.138225 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138222    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="init-config-reloader"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138228    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="init-config-reloader"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138237    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-web"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138243    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-web"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138249    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="thanos-sidecar"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138254    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="thanos-sidecar"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138260    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-thanos"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138265    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-thanos"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138315    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="prometheus"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138325    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-web"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138330    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138336    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="thanos-sidecar"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138343    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="config-reloader"
Sep 18 19:44:32.138433 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.138348    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" containerName="kube-rbac-proxy-thanos"
Sep 18 19:44:32.142028 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.142013    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.144226 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144207    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:44:32.144319 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144223    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:44:32.144319 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144231    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7hhm6\""
Sep 18 19:44:32.144319 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144239    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:44:32.144518 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144506    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:44:32.144764 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:44:32.144892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:44:32.144892 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144804    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:44:32.145018 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.144995    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:44:32.145495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.145467    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cnnd0p2jiu0e9\""
Sep 18 19:44:32.145495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.145475    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:44:32.145495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.145480    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:44:32.145709 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.145541    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:44:32.145781 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.145765    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:44:32.148251 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.148235    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:44:32.150660 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.150639    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:44:32.157076 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.157056    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:32.219016 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.218993    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vdv\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-kube-api-access-89vdv\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219105 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219105 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219105 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219240 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219240 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219159    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219240 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219240 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219240 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219226    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219526 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219511    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.219785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.219646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320246 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320221    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320457    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320571 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320559    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320618    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vdv\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-kube-api-access-89vdv\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.320853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.321241 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.320854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.321241 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.321005    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.321241 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.321057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.321530 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.321508    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.321668 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.321650    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.322087 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.322068    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.322170 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.322158    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.322313 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.322294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.323535 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.323346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.323747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.323722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.323932 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.323892    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.324187 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.324163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.324289 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.324187    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.324594 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.324572    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.325060 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.325027    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.325745 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.325718    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-config\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.326025 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.326008    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.326235 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.326217    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.326325 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.326308    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.326441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.326425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.330802 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.330778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vdv\" (UniqueName: \"kubernetes.io/projected/5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6-kube-api-access-89vdv\") pod \"prometheus-k8s-0\" (UID: \"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.352992 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.352941    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46dd894c-5d68-4f63-81b4-ec0ddc830acc" path="/var/lib/kubelet/pods/46dd894c-5d68-4f63-81b4-ec0ddc830acc/volumes"
Sep 18 19:44:32.452017 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.451964    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:32.736080 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.736049    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:44:32.738680 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:44:32.738651    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1e08a4_21bd_49d1_aef7_94d2d7f5d2f6.slice/crio-9bfff778b960802f5f3830a054bf4f2aff0c6a3bb709f91e77b3895554e3b2a9 WatchSource:0}: Error finding container 9bfff778b960802f5f3830a054bf4f2aff0c6a3bb709f91e77b3895554e3b2a9: Status 404 returned error can't find the container with id 9bfff778b960802f5f3830a054bf4f2aff0c6a3bb709f91e77b3895554e3b2a9
Sep 18 19:44:32.794674 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:32.794647    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"9bfff778b960802f5f3830a054bf4f2aff0c6a3bb709f91e77b3895554e3b2a9"}
Sep 18 19:44:33.806175 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:33.806142    2567 generic.go:356] "Generic (PLEG): container finished" podID="5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6" containerID="b448f4b48f0c98dadabfec4824e208781f717dcc337994ddd7cb30afe4c26745" exitCode=0
Sep 18 19:44:33.806570 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:33.806190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerDied","Data":"b448f4b48f0c98dadabfec4824e208781f717dcc337994ddd7cb30afe4c26745"}
Sep 18 19:44:34.813734 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813692    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"34a05684900879b76723a5f1a0dd95f4530c5f0c68f9849d68e3c9baf1594ddb"}
Sep 18 19:44:34.814102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"b5d40e02853511a29581c8939adbfd43e14993958993fc706252cd25f9ab57b4"}
Sep 18 19:44:34.814102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813793    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"7315441ed78b74beea6d85b8d14d0f6efcbd1c215e3cc8456468046cf8c5fcb1"}
Sep 18 19:44:34.814102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"88387afc1aee6268afb309bac9d92011e408e0d6a8a296b8f763ea79779ae182"}
Sep 18 19:44:34.814102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813843    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"074422c0bc22130464eb4fce7fddb42205e1b924b5291c32b1f56f6fd7f1b253"}
Sep 18 19:44:34.814102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.813865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6","Type":"ContainerStarted","Data":"f12b391a4955d4f314690c162a5f66ac5c66e4d74a639f7c356abe1ccc765a01"}
Sep 18 19:44:34.838932 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:34.838884    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.819504876 podStartE2EDuration="2.83886902s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.364144ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:34.837435634 +0000 UTC m=+372.616806215" watchObservedRunningTime="2026-09-18 19:44:34.83886902 +0000 UTC m=+372.618239602"
Sep 18 19:44:37.453015 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:37.452981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:44:38.270402 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.270341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:44:38.272612 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.272586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695e01df-28bb-4aed-9e24-d53720c990fc-metrics-certs\") pod \"network-metrics-daemon-m6p9z\" (UID: \"695e01df-28bb-4aed-9e24-d53720c990fc\") " pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:44:38.450512 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.450482    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nv7t\""
Sep 18 19:44:38.459041 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.459026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6p9z"
Sep 18 19:44:38.728158 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.728131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6p9z"]
Sep 18 19:44:38.730862 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:44:38.730829    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e01df_28bb_4aed_9e24_d53720c990fc.slice/crio-7ef1bec8c1b00ff459564bb4945d824c603c3873c7c832f45e531fd7017f31bb WatchSource:0}: Error finding container 7ef1bec8c1b00ff459564bb4945d824c603c3873c7c832f45e531fd7017f31bb: Status 404 returned error can't find the container with id 7ef1bec8c1b00ff459564bb4945d824c603c3873c7c832f45e531fd7017f31bb
Sep 18 19:44:38.733194 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.733181    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:44:38.833568 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:38.833538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6p9z" event={"ID":"695e01df-28bb-4aed-9e24-d53720c990fc","Type":"ContainerStarted","Data":"7ef1bec8c1b00ff459564bb4945d824c603c3873c7c832f45e531fd7017f31bb"}
Sep 18 19:44:39.839672 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:39.839650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6p9z" event={"ID":"695e01df-28bb-4aed-9e24-d53720c990fc","Type":"ContainerStarted","Data":"e81216958521dc4a2ba080e8dddcbc666befd9c2f8f690e54ff4dfc38fba8f86"}
Sep 18 19:44:40.845500 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:40.845469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6p9z" event={"ID":"695e01df-28bb-4aed-9e24-d53720c990fc","Type":"ContainerStarted","Data":"a9b8b2d04c1f0797929f1b69cf1bd4162aabf1d6174fff3df77025378728a576"}
Sep 18 19:44:40.860316 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:44:40.860275    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m6p9z" podStartSLOduration=375.913880176 podStartE2EDuration="6m16.860261523s" totalImagesPullingTime="946.381347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:40.859067545 +0000 UTC m=+378.638438126" watchObservedRunningTime="2026-09-18 19:44:40.860261523 +0000 UTC m=+378.639632170"
Sep 18 19:45:32.452657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:45:32.452601    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:32.468455 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:45:32.468420    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:45:33.074522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:45:33.074492    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:48:24.240785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:24.240757    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:48:24.241573 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:24.241557    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:48:35.398765 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.398732    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6vt92"]
Sep 18 19:48:35.401941 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.401919    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.408500 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.408483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:48:35.408775 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.408761    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:48:35.409613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.409595    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:48:35.412082 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.412068    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-l6bqs\""
Sep 18 19:48:35.412642 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.412625    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:48:35.425241 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.425218    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6vt92"]
Sep 18 19:48:35.425581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.425549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.425658 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.425598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpppx\" (UniqueName: \"kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.526633 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.526604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.526777 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.526639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpppx\" (UniqueName: \"kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.526996 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.526976    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.535196 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.535179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpppx\" (UniqueName: \"kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx\") pod \"s3-init-6vt92\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") " pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.721819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.721751    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6vt92"
Sep 18 19:48:35.993237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:35.993165    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6vt92"]
Sep 18 19:48:35.998267 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:48:35.998236    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01b9bebe_d3f5_4fe8_ab78_3b36d55eb802.slice/crio-10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f WatchSource:0}: Error finding container 10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f: Status 404 returned error can't find the container with id 10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f
Sep 18 19:48:36.795897 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:36.795859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerStarted","Data":"10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f"}
Sep 18 19:48:40.816543 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:40.816513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerStarted","Data":"69dbeec5a00fd6c62965cc0f738ef61e6bc2f498b53ff4906aa0f2a98077f246"}
Sep 18 19:48:42.826819 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:42.826785    2567 generic.go:356] "Generic (PLEG): container finished" podID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerID="69dbeec5a00fd6c62965cc0f738ef61e6bc2f498b53ff4906aa0f2a98077f246" exitCode=0
Sep 18 19:48:42.827215 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:42.826846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerDied","Data":"69dbeec5a00fd6c62965cc0f738ef61e6bc2f498b53ff4906aa0f2a98077f246"}
Sep 18 19:48:47.858270 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:47.858226    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerStarted","Data":"f2ba2df5757ab92b8bba25af69b225ecdf894e1dc59e22022bf3f871419e3ff5"}
Sep 18 19:48:47.878680 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:48:47.878621    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6vt92" podStartSLOduration=2.6466947899999997 podStartE2EDuration="12.878600421s" totalImagesPullingTime="8.462667019s" totalInitContainerRuntime="1.769238612s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:47.875596352 +0000 UTC m=+625.654966933" watchObservedRunningTime="2026-09-18 19:48:47.878600421 +0000 UTC m=+625.657971211"
Sep 18 19:49:52.118993 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:52.118960    2567 generic.go:356] "Generic (PLEG): container finished" podID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerID="f2ba2df5757ab92b8bba25af69b225ecdf894e1dc59e22022bf3f871419e3ff5" exitCode=0
Sep 18 19:49:52.119353 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:52.119033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerDied","Data":"f2ba2df5757ab92b8bba25af69b225ecdf894e1dc59e22022bf3f871419e3ff5"}
Sep 18 19:49:53.251466 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.251446    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6vt92"
Sep 18 19:49:53.354898 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.354868    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache\") pod \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") "
Sep 18 19:49:53.355050 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.354935    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpppx\" (UniqueName: \"kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx\") pod \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\" (UID: \"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802\") "
Sep 18 19:49:53.357079 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.357058    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx" (OuterVolumeSpecName: "kube-api-access-bpppx") pod "01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" (UID: "01b9bebe-d3f5-4fe8-ab78-3b36d55eb802"). InnerVolumeSpecName "kube-api-access-bpppx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:49:53.373780 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.373728    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache" (OuterVolumeSpecName: "model-cache") pod "01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" (UID: "01b9bebe-d3f5-4fe8-ab78-3b36d55eb802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:53.456395 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.456351    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpppx\" (UniqueName: \"kubernetes.io/projected/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-kube-api-access-bpppx\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:49:53.456487 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:53.456400    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01b9bebe-d3f5-4fe8-ab78-3b36d55eb802-model-cache\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:49:54.128978 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:54.128955    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6vt92"
Sep 18 19:49:54.129135 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:54.128980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6vt92" event={"ID":"01b9bebe-d3f5-4fe8-ab78-3b36d55eb802","Type":"ContainerDied","Data":"10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f"}
Sep 18 19:49:54.129135 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:49:54.129012    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10874568862fdb4f8e210ee4cc6e07a72caccc070b8cf40cdb705058d318dc0f"
Sep 18 19:50:06.212938 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.212906    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:50:06.213557 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.213487    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerName="s3-init"
Sep 18 19:50:06.213557 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.213507    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerName="s3-init"
Sep 18 19:50:06.213557 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.213527    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerName="download-hf-model"
Sep 18 19:50:06.213557 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.213536    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerName="download-hf-model"
Sep 18 19:50:06.213792 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.213629    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b9bebe-d3f5-4fe8-ab78-3b36d55eb802" containerName="s3-init"
Sep 18 19:50:06.217205 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.217184    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.219918 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.219861    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-9dxnf\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\"" type="*v1.Secret"
Sep 18 19:50:06.219918 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.219904    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap"
Sep 18 19:50:06.220080 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.219953    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-canary-rollback-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" type="*v1.Secret"
Sep 18 19:50:06.220147 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.220126    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.220204 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.220140    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Sep 18 19:50:06.220204 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.220171    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 18 19:50:06.220204 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.220189    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 18 19:50:06.220449 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:06.220432    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-canary-rollback-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 18 19:50:06.247113 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.247065    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:50:06.314704 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.314669    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:50:06.318152 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.318135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.321522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.321497    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b7ffd-kube-rbac-proxy-sar-config\""
Sep 18 19:50:06.321522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.321497    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b7ffd-predictor-serving-cert\""
Sep 18 19:50:06.342466 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.342435    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:50:06.363328 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.363290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.363514 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.363334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.363514 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.363387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.363514 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.363431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.363514 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.363452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp5h\" (UniqueName: \"kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464635 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464635 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp5h\" (UniqueName: \"kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464635 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464604    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464660    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464709    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464810    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.464875 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.464838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcls\" (UniqueName: \"kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.465109 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.465059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:06.566156 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566124    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566170    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566327 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcls\" (UniqueName: \"kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566491 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566755 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.566896 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.566879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:06.568697 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:06.568677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:07.066667 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.066633    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:50:07.203414 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.203357    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:50:07.205483 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.205463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:07.207561 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.207539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:07.407390 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.407294    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:50:07.414685 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.414659    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp5h\" (UniqueName: \"kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:07.414843 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.414714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcls\" (UniqueName: \"kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls\") pod \"isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:07.418315 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.418294    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 19:50:07.465735 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:07.465712    2567 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-canary-rollback-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 18 19:50:07.465866 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:07.465784    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config podName:7f178a42-ce5a-47cd-a20c-67658eaed665 nodeName:}" failed. No retries permitted until 2026-09-18 19:50:07.96576444 +0000 UTC m=+705.745135003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config") pod "isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665") : failed to sync configmap cache: timed out waiting for the condition
Sep 18 19:50:07.465866 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:07.465710    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 18 19:50:07.465994 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:50:07.465871    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls podName:7f178a42-ce5a-47cd-a20c-67658eaed665 nodeName:}" failed. No retries permitted until 2026-09-18 19:50:07.96585461 +0000 UTC m=+705.745225170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665") : failed to sync secret cache: timed out waiting for the condition
Sep 18 19:50:07.528377 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.528350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:07.546760 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.546741    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:50:07.683345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.683262    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:50:07.752766 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.752740    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:50:07.800202 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.800177    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:50:07.802127 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:50:07.802103    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046aaf90_8702_48bc_888b_84e640067819.slice/crio-691e4ebc5f625eba45c746754a36880aebc6808538ca81c2864584dfc5648325 WatchSource:0}: Error finding container 691e4ebc5f625eba45c746754a36880aebc6808538ca81c2864584dfc5648325: Status 404 returned error can't find the container with id 691e4ebc5f625eba45c746754a36880aebc6808538ca81c2864584dfc5648325
Sep 18 19:50:07.979814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.979726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:07.979814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.979777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:07.980531 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.980513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:07.982255 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:07.982240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9d4gz\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:08.027871 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:08.027834    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:08.189436 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:08.189389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerStarted","Data":"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"}
Sep 18 19:50:08.189638 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:08.189459    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerStarted","Data":"691e4ebc5f625eba45c746754a36880aebc6808538ca81c2864584dfc5648325"}
Sep 18 19:50:08.306149 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:08.306120    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:50:08.308446 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:50:08.308420    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f178a42_ce5a_47cd_a20c_67658eaed665.slice/crio-3540bb938635c7dbc8b9a23993c9a13e49a49a0899e244b4130157b796bfcc68 WatchSource:0}: Error finding container 3540bb938635c7dbc8b9a23993c9a13e49a49a0899e244b4130157b796bfcc68: Status 404 returned error can't find the container with id 3540bb938635c7dbc8b9a23993c9a13e49a49a0899e244b4130157b796bfcc68
Sep 18 19:50:09.196145 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:09.196107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerStarted","Data":"1d6f9d8797b1e1d425174cf02a85aed1c23d2ec7297e41967eb1e4f019078d7a"}
Sep 18 19:50:09.196634 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:09.196177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerStarted","Data":"3540bb938635c7dbc8b9a23993c9a13e49a49a0899e244b4130157b796bfcc68"}
Sep 18 19:50:12.211494 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:12.211460    2567 generic.go:356] "Generic (PLEG): container finished" podID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerID="1d6f9d8797b1e1d425174cf02a85aed1c23d2ec7297e41967eb1e4f019078d7a" exitCode=0
Sep 18 19:50:12.211965 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:12.211539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerDied","Data":"1d6f9d8797b1e1d425174cf02a85aed1c23d2ec7297e41967eb1e4f019078d7a"}
Sep 18 19:50:12.212868 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:12.212852    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:50:12.213300 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:12.213276    2567 generic.go:356] "Generic (PLEG): container finished" podID="046aaf90-8702-48bc-888b-84e640067819" containerID="23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee" exitCode=0
Sep 18 19:50:12.213392 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:12.213332    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerDied","Data":"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"}
Sep 18 19:50:28.328144 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:28.328110    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerStarted","Data":"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"}
Sep 18 19:50:28.329788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:28.329766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerStarted","Data":"3f6dcdc418bc754540cc041e6d8c92d5cb10a5a3de4f4288380b8b4717b46d09"}
Sep 18 19:50:31.349031 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:31.348997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerStarted","Data":"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"}
Sep 18 19:50:31.351037 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:31.351015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerStarted","Data":"0e3b02342d628844ca2e95ea7f657a2e7c80e3ebdca44b83967e889002126603"}
Sep 18 19:50:31.351192 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:31.351175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:31.369193 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:31.369127    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podStartSLOduration=3.049961295 podStartE2EDuration="25.369106828s" totalImagesPullingTime="18.645740229s" totalInitContainerRuntime="3.673405304s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:31.367781553 +0000 UTC m=+729.147152147" watchObservedRunningTime="2026-09-18 19:50:31.369106828 +0000 UTC m=+729.148477411"
Sep 18 19:50:32.358536 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:32.358503    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:32.359937 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:32.359876    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:50:33.363415 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:33.363379    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:50:34.369954 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.369917    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerStarted","Data":"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"}
Sep 18 19:50:34.370420 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.370156    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:34.370420 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.370215    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:34.370420 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.370250    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:34.371539 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.371508    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:50:34.372081 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.372062    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:34.392495 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:34.392439    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podStartSLOduration=3.43117096 podStartE2EDuration="28.39241804s" totalImagesPullingTime="21.079779764s" totalInitContainerRuntime="3.881467316s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:34.390718073 +0000 UTC m=+732.170088659" watchObservedRunningTime="2026-09-18 19:50:34.39241804 +0000 UTC m=+732.171788621"
Sep 18 19:50:35.374883 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:35.374838    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:50:35.375351 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:35.375219    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:35.378838 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:35.378819    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:50:36.379140 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:36.379097    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:50:36.379556 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:36.379378    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:38.375062 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:38.375036    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:50:38.375613 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:38.375584    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:50:46.379650 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:46.379595    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:50:46.380066 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:46.380013    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:48.375594 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:48.375557    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:50:56.379998 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:56.379953    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:50:56.380509 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:56.380426    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:58.375728 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:50:58.375684    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:51:06.379679 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:06.379622    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:51:06.380224 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:06.380151    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:08.375810 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:08.375770    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:51:16.379856 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:16.379807    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:51:16.380351 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:16.380286    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:18.375699 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:18.375662    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:51:26.379717 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:26.379663    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:51:26.380153 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:26.380126    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:28.375897 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:28.375858    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:51:36.379206 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:36.379100    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:51:36.379614 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:36.379582    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:38.376132 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:38.376096    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 18 19:51:45.348547 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:45.348514    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:51:46.379660 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:46.379633    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:51:46.380101 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:51:46.380053    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:00.961459 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:00.961429    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:52:00.961953 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:00.961725    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" containerID="cri-o://ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b" gracePeriod=30
Sep 18 19:52:00.961953 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:00.961774    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" containerID="cri-o://eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341" gracePeriod=30
Sep 18 19:52:00.961953 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:00.961735    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" containerID="cri-o://24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0" gracePeriod=30
Sep 18 19:52:01.782410 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:01.782353    2567 generic.go:356] "Generic (PLEG): container finished" podID="046aaf90-8702-48bc-888b-84e640067819" containerID="eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341" exitCode=2
Sep 18 19:52:01.782580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:01.782413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerDied","Data":"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"}
Sep 18 19:52:05.374996 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:05.374956    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:05.804029 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:05.803995    2567 generic.go:356] "Generic (PLEG): container finished" podID="046aaf90-8702-48bc-888b-84e640067819" containerID="ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b" exitCode=0
Sep 18 19:52:05.804187 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:05.804066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerDied","Data":"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"}
Sep 18 19:52:06.379438 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:06.379402    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:06.380591 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:06.380561    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:52:10.375626 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:10.375591    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:15.375821 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:15.375778    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:15.376238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:15.375910    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:16.379585 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:16.379548    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:16.380022 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:16.379999    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:52:20.375116 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:20.375080    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:25.375191 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:25.375148    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:26.379923 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:26.379884    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:26.380385 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:26.380024    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 18 19:52:26.380385 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:26.380061    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:26.380385 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:26.380175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:30.375139 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:30.375103    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 18 19:52:31.137873 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.137853    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:31.149542 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149519    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzcls\" (UniqueName: \"kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls\") pod \"046aaf90-8702-48bc-888b-84e640067819\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") "
Sep 18 19:52:31.149657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149554    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config\") pod \"046aaf90-8702-48bc-888b-84e640067819\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") "
Sep 18 19:52:31.149657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149592    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert\") pod \"046aaf90-8702-48bc-888b-84e640067819\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") "
Sep 18 19:52:31.149657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149624    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls\") pod \"046aaf90-8702-48bc-888b-84e640067819\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") "
Sep 18 19:52:31.149657 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location\") pod \"046aaf90-8702-48bc-888b-84e640067819\" (UID: \"046aaf90-8702-48bc-888b-84e640067819\") "
Sep 18 19:52:31.149970 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149949    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "046aaf90-8702-48bc-888b-84e640067819" (UID: "046aaf90-8702-48bc-888b-84e640067819"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:31.150029 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.149966    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "046aaf90-8702-48bc-888b-84e640067819" (UID: "046aaf90-8702-48bc-888b-84e640067819"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:31.150029 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.150008    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "046aaf90-8702-48bc-888b-84e640067819" (UID: "046aaf90-8702-48bc-888b-84e640067819"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:31.151670 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.151645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls" (OuterVolumeSpecName: "kube-api-access-qzcls") pod "046aaf90-8702-48bc-888b-84e640067819" (UID: "046aaf90-8702-48bc-888b-84e640067819"). InnerVolumeSpecName "kube-api-access-qzcls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:31.151750 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.151708    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "046aaf90-8702-48bc-888b-84e640067819" (UID: "046aaf90-8702-48bc-888b-84e640067819"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:31.250522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.250463    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:52:31.250522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.250487    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046aaf90-8702-48bc-888b-84e640067819-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:52:31.250522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.250497    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046aaf90-8702-48bc-888b-84e640067819-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:52:31.250522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.250506    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcls\" (UniqueName: \"kubernetes.io/projected/046aaf90-8702-48bc-888b-84e640067819-kube-api-access-qzcls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:52:31.250522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.250515    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046aaf90-8702-48bc-888b-84e640067819-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:52:31.926242 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.926209    2567 generic.go:356] "Generic (PLEG): container finished" podID="046aaf90-8702-48bc-888b-84e640067819" containerID="24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0" exitCode=0
Sep 18 19:52:31.926673 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.926271    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"
Sep 18 19:52:31.926673 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.926295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerDied","Data":"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"}
Sep 18 19:52:31.926673 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.926361    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w" event={"ID":"046aaf90-8702-48bc-888b-84e640067819","Type":"ContainerDied","Data":"691e4ebc5f625eba45c746754a36880aebc6808538ca81c2864584dfc5648325"}
Sep 18 19:52:31.926673 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.926396    2567 scope.go:117] "RemoveContainer" containerID="24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"
Sep 18 19:52:31.936107 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.936092    2567 scope.go:117] "RemoveContainer" containerID="eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"
Sep 18 19:52:31.943409 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.943395    2567 scope.go:117] "RemoveContainer" containerID="ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"
Sep 18 19:52:31.949493 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.949475    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:52:31.950790 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.950771    2567 scope.go:117] "RemoveContainer" containerID="23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"
Sep 18 19:52:31.951996 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.951977    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b7ffd-predictor-588fd9b74c-w4r5w"]
Sep 18 19:52:31.958002 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.957987    2567 scope.go:117] "RemoveContainer" containerID="24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"
Sep 18 19:52:31.958251 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:52:31.958235    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0\": container with ID starting with 24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0 not found: ID does not exist" containerID="24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"
Sep 18 19:52:31.958295 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958258    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0"} err="failed to get container status \"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0\": rpc error: code = NotFound desc = could not find container \"24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0\": container with ID starting with 24b6ae3f5d59c0779cc4ef76b67d8118ef4daeb50bc1e31ed78821270477ede0 not found: ID does not exist"
Sep 18 19:52:31.958295 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958273    2567 scope.go:117] "RemoveContainer" containerID="eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"
Sep 18 19:52:31.958531 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:52:31.958514    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341\": container with ID starting with eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341 not found: ID does not exist" containerID="eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"
Sep 18 19:52:31.958582 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958536    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341"} err="failed to get container status \"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341\": rpc error: code = NotFound desc = could not find container \"eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341\": container with ID starting with eb6056823e788f6ef5e895e7f5eb4a14a4df1070914154ae995ff8845fe4a341 not found: ID does not exist"
Sep 18 19:52:31.958582 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958549    2567 scope.go:117] "RemoveContainer" containerID="ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"
Sep 18 19:52:31.958794 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:52:31.958773    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b\": container with ID starting with ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b not found: ID does not exist" containerID="ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"
Sep 18 19:52:31.958841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958797    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b"} err="failed to get container status \"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b\": rpc error: code = NotFound desc = could not find container \"ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b\": container with ID starting with ce0b17d1dc8519e7ad5193211b306b1c21a9e9ca5731005937a6a5661b611a5b not found: ID does not exist"
Sep 18 19:52:31.958841 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.958809    2567 scope.go:117] "RemoveContainer" containerID="23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"
Sep 18 19:52:31.959016 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:52:31.959001    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee\": container with ID starting with 23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee not found: ID does not exist" containerID="23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"
Sep 18 19:52:31.959055 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:31.959020    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee"} err="failed to get container status \"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee\": rpc error: code = NotFound desc = could not find container \"23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee\": container with ID starting with 23e711b1971200648f0d1b52f209f81a8157337de91c43b51241fa0c3d15c6ee not found: ID does not exist"
Sep 18 19:52:32.353402 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:52:32.353356    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046aaf90-8702-48bc-888b-84e640067819" path="/var/lib/kubelet/pods/046aaf90-8702-48bc-888b-84e640067819/volumes"
Sep 18 19:53:24.270660 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:24.270585    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:53:24.272931 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:24.272904    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:53:26.072349 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:26.072316    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:53:26.072835 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:26.072579    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container" containerID="cri-o://3f6dcdc418bc754540cc041e6d8c92d5cb10a5a3de4f4288380b8b4717b46d09" gracePeriod=30
Sep 18 19:53:26.072835 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:26.072612    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kube-rbac-proxy" containerID="cri-o://0e3b02342d628844ca2e95ea7f657a2e7c80e3ebdca44b83967e889002126603" gracePeriod=30
Sep 18 19:53:27.175878 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:27.175843    2567 generic.go:356] "Generic (PLEG): container finished" podID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerID="0e3b02342d628844ca2e95ea7f657a2e7c80e3ebdca44b83967e889002126603" exitCode=2
Sep 18 19:53:27.176249 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:27.175915    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerDied","Data":"0e3b02342d628844ca2e95ea7f657a2e7c80e3ebdca44b83967e889002126603"}
Sep 18 19:53:28.364272 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:28.364238    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 18 19:53:30.191300 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.191273    2567 generic.go:356] "Generic (PLEG): container finished" podID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerID="3f6dcdc418bc754540cc041e6d8c92d5cb10a5a3de4f4288380b8b4717b46d09" exitCode=0
Sep 18 19:53:30.191637 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.191343    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerDied","Data":"3f6dcdc418bc754540cc041e6d8c92d5cb10a5a3de4f4288380b8b4717b46d09"}
Sep 18 19:53:30.312337 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.312313    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:53:30.439064 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.438998    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location\") pod \"7f178a42-ce5a-47cd-a20c-67658eaed665\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") "
Sep 18 19:53:30.439064 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439034    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert\") pod \"7f178a42-ce5a-47cd-a20c-67658eaed665\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") "
Sep 18 19:53:30.439238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439099    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") pod \"7f178a42-ce5a-47cd-a20c-67658eaed665\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") "
Sep 18 19:53:30.439238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439118    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbp5h\" (UniqueName: \"kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h\") pod \"7f178a42-ce5a-47cd-a20c-67658eaed665\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") "
Sep 18 19:53:30.439238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439144    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") pod \"7f178a42-ce5a-47cd-a20c-67658eaed665\" (UID: \"7f178a42-ce5a-47cd-a20c-67658eaed665\") "
Sep 18 19:53:30.439400 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439350    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f178a42-ce5a-47cd-a20c-67658eaed665" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:30.439504 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439473    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f178a42-ce5a-47cd-a20c-67658eaed665" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:30.439597 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.439555    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f178a42-ce5a-47cd-a20c-67658eaed665" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:30.441226 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.441200    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h" (OuterVolumeSpecName: "kube-api-access-wbp5h") pod "7f178a42-ce5a-47cd-a20c-67658eaed665" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665"). InnerVolumeSpecName "kube-api-access-wbp5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:30.441320 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.441252    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f178a42-ce5a-47cd-a20c-67658eaed665" (UID: "7f178a42-ce5a-47cd-a20c-67658eaed665"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:30.540102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.540064    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbp5h\" (UniqueName: \"kubernetes.io/projected/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-api-access-wbp5h\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.540102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.540101    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.540102 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.540111    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f178a42-ce5a-47cd-a20c-67658eaed665-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.540305 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.540122    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f178a42-ce5a-47cd-a20c-67658eaed665-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.540305 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:30.540132    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f178a42-ce5a-47cd-a20c-67658eaed665-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:53:31.197788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.197750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz" event={"ID":"7f178a42-ce5a-47cd-a20c-67658eaed665","Type":"ContainerDied","Data":"3540bb938635c7dbc8b9a23993c9a13e49a49a0899e244b4130157b796bfcc68"}
Sep 18 19:53:31.197788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.197775    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"
Sep 18 19:53:31.198323 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.197816    2567 scope.go:117] "RemoveContainer" containerID="0e3b02342d628844ca2e95ea7f657a2e7c80e3ebdca44b83967e889002126603"
Sep 18 19:53:31.207401 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.207381    2567 scope.go:117] "RemoveContainer" containerID="3f6dcdc418bc754540cc041e6d8c92d5cb10a5a3de4f4288380b8b4717b46d09"
Sep 18 19:53:31.217035 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.217006    2567 scope.go:117] "RemoveContainer" containerID="1d6f9d8797b1e1d425174cf02a85aed1c23d2ec7297e41967eb1e4f019078d7a"
Sep 18 19:53:31.223877 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.223853    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:53:31.230611 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.230584    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9d4gz"]
Sep 18 19:53:31.324873 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.324842    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:53:31.325345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325323    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent"
Sep 18 19:53:31.325345 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325342    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325376    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="storage-initializer"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325385    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="storage-initializer"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325408    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="storage-initializer"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325417    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="storage-initializer"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325427    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325435    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325445    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325453    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325470    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325479    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325497    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container"
Sep 18 19:53:31.325537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325505    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container"
Sep 18 19:53:31.325940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325580    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325591    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kserve-container"
Sep 18 19:53:31.325940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325601    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="kserve-container"
Sep 18 19:53:31.325940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325608    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" containerName="kube-rbac-proxy"
Sep 18 19:53:31.325940 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.325616    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="046aaf90-8702-48bc-888b-84e640067819" containerName="agent"
Sep 18 19:53:31.328976 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.328959    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.331686 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.331665    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:53:31.331805 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.331703    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:53:31.331805 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.331760    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:53:31.332099 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.332072    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:53:31.332198 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.332116    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 19:53:31.332198 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.332129    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:53:31.332198 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.332120    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:53:31.340738 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.340716    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:53:31.449563 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.449477    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qlx\" (UniqueName: \"kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.449563 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.449549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.449801 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.449597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.449801 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.449632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.449801 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.449704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550097 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550063    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qlx\" (UniqueName: \"kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550275 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550275 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550275 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550192    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550275 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550515 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:53:31.550317    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:53:31.550515 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:53:31.550408    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls podName:310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54 nodeName:}" failed. No retries permitted until 2026-09-18 19:53:32.050386496 +0000 UTC m=+909.829757061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls") pod "isvc-canary-stop-predictor-5d5b899946-mhxmz" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:53:31.550681 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550935 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.550992 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.550944    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:31.558441 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:31.558419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qlx\" (UniqueName: \"kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:32.055951 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:32.055918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:32.058339 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:32.058312    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-mhxmz\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:32.239953 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:32.239925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:32.353202 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:32.353127    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f178a42-ce5a-47cd-a20c-67658eaed665" path="/var/lib/kubelet/pods/7f178a42-ce5a-47cd-a20c-67658eaed665/volumes"
Sep 18 19:53:32.512100 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:32.512042    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:53:32.514256 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:53:32.514229    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310f5d1e_6c8b_47c0_aaf2_b8b04c6eed54.slice/crio-071bcb9231d6033a0d789aad409b92f41c64f2d0f9bfe36c0cd57185d38e1654 WatchSource:0}: Error finding container 071bcb9231d6033a0d789aad409b92f41c64f2d0f9bfe36c0cd57185d38e1654: Status 404 returned error can't find the container with id 071bcb9231d6033a0d789aad409b92f41c64f2d0f9bfe36c0cd57185d38e1654
Sep 18 19:53:33.210081 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:33.210046    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerStarted","Data":"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"}
Sep 18 19:53:33.211219 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:33.210127    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerStarted","Data":"071bcb9231d6033a0d789aad409b92f41c64f2d0f9bfe36c0cd57185d38e1654"}
Sep 18 19:53:36.225226 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:36.225197    2567 generic.go:356] "Generic (PLEG): container finished" podID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerID="fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2" exitCode=0
Sep 18 19:53:36.225538 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:36.225235    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerDied","Data":"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"}
Sep 18 19:53:37.231421 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.231390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerStarted","Data":"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"}
Sep 18 19:53:37.231783 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.231450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerStarted","Data":"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"}
Sep 18 19:53:37.231848 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.231781    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:37.231848 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.231841    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:37.233244 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.233218    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:53:37.252395 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:37.252338    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podStartSLOduration=2.665195561 podStartE2EDuration="6.252325103s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.587129542s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:37.251417457 +0000 UTC m=+915.030788039" watchObservedRunningTime="2026-09-18 19:53:37.252325103 +0000 UTC m=+915.031695684"
Sep 18 19:53:38.236850 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:38.236807    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:53:43.241393 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:43.241334    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:53:43.241969 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:43.241943    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:53:51.466656 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.466627    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:53:51.469230 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.469212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.471449 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.471427    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e0026-kube-rbac-proxy-sar-config\""
Sep 18 19:53:51.471560 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.471468    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e0026-predictor-serving-cert\""
Sep 18 19:53:51.483090 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.483060    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:53:51.571698 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.571662    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:53:51.574108 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.574091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.576328 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.576303    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e0026-predictor-serving-cert\""
Sep 18 19:53:51.576328 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.576323    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e0026-kube-rbac-proxy-sar-config\""
Sep 18 19:53:51.586890 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.586870    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:53:51.612303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.612279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5wm\" (UniqueName: \"kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.612416 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.612324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.612470 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.612428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.612510 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.612480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.612548 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.612514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.713632 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.713733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.713733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.713808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5wm\" (UniqueName: \"kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.713808 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.713870 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.713903 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.713939 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.713973 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713961    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.714021 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.713992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vz5\" (UniqueName: \"kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.714347 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.714326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.714585 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.714568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.714634 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.714586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.716312 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.716288    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.722082 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.722030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5wm\" (UniqueName: \"kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm\") pod \"isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.787141 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.787109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:51.815074 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815184 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815184 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815184 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815344 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vz5\" (UniqueName: \"kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815344 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:53:51.815252    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-e0026-predictor-serving-cert" not found
Sep 18 19:53:51.815471 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:53:51.815360    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls podName:faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd nodeName:}" failed. No retries permitted until 2026-09-18 19:53:52.315335246 +0000 UTC m=+930.094705820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd") : secret "isvc-xgboost-graph-raw-hpa-e0026-predictor-serving-cert" not found
Sep 18 19:53:51.815895 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815870    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.815980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.815940    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:51.823237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:51.823207    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vz5\" (UniqueName: \"kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:52.056844 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.056820    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:53:52.059030 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:53:52.059005    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bcc744_7e00_4299_950c_59018f18ecc6.slice/crio-8f136be2c4d2ca2b14b3195f96567db2e8cbd047fbb4227dbab3f300ecbdd06b WatchSource:0}: Error finding container 8f136be2c4d2ca2b14b3195f96567db2e8cbd047fbb4227dbab3f300ecbdd06b: Status 404 returned error can't find the container with id 8f136be2c4d2ca2b14b3195f96567db2e8cbd047fbb4227dbab3f300ecbdd06b
Sep 18 19:53:52.309132 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.309045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerStarted","Data":"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"}
Sep 18 19:53:52.309132 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.309114    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerStarted","Data":"8f136be2c4d2ca2b14b3195f96567db2e8cbd047fbb4227dbab3f300ecbdd06b"}
Sep 18 19:53:52.320136 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.320109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:52.322358 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.322341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:52.483947 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.483918    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:53:52.757762 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:52.757738    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:53:52.759132 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:53:52.759103    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf2e9f7_e654_4e5b_89e1_ddb348f7d0bd.slice/crio-f7ab76248102806c00d5bd5a3f10a0232ec0dcca42142bef7c1f0ffed652dc23 WatchSource:0}: Error finding container f7ab76248102806c00d5bd5a3f10a0232ec0dcca42142bef7c1f0ffed652dc23: Status 404 returned error can't find the container with id f7ab76248102806c00d5bd5a3f10a0232ec0dcca42142bef7c1f0ffed652dc23
Sep 18 19:53:53.242410 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:53.242356    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:53:53.315746 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:53.315708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerStarted","Data":"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"}
Sep 18 19:53:53.315923 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:53.315768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerStarted","Data":"f7ab76248102806c00d5bd5a3f10a0232ec0dcca42142bef7c1f0ffed652dc23"}
Sep 18 19:53:56.331662 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:56.331628    2567 generic.go:356] "Generic (PLEG): container finished" podID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerID="a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff" exitCode=0
Sep 18 19:53:56.332038 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:56.331698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerDied","Data":"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"}
Sep 18 19:53:57.338311 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.338272    2567 generic.go:356] "Generic (PLEG): container finished" podID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerID="7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154" exitCode=0
Sep 18 19:53:57.338752 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.338346    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerDied","Data":"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"}
Sep 18 19:53:57.341210 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.341183    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerStarted","Data":"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"}
Sep 18 19:53:57.341281 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.341251    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerStarted","Data":"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"}
Sep 18 19:53:57.341501 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.341483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:57.374733 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:57.374691    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podStartSLOduration=2.75107376 podStartE2EDuration="6.374676125s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.623602365s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:57.372976062 +0000 UTC m=+935.152346634" watchObservedRunningTime="2026-09-18 19:53:57.374676125 +0000 UTC m=+935.154046707"
Sep 18 19:53:58.350400 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:58.350088    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:53:58.356018 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:58.355990    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:53:59.355844 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:53:59.355797    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:03.242117 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:03.242079    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:54:04.360765 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:04.360735    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:54:04.361340 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:04.361302    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:13.241918 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:13.241859    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:54:14.361655 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:14.361609    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:17.453583 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.453552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerStarted","Data":"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"}
Sep 18 19:54:17.454006 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.453612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerStarted","Data":"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"}
Sep 18 19:54:17.454006 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.453847    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:54:17.454006 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.453910    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:54:17.455396 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.455353    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:17.472594 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:17.472550    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podStartSLOduration=3.052401089 podStartE2EDuration="26.472516189s" totalImagesPullingTime="19.575741674s" totalInitContainerRuntime="3.844373426s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:17.471564998 +0000 UTC m=+955.250935580" watchObservedRunningTime="2026-09-18 19:54:17.472516189 +0000 UTC m=+955.251886771"
Sep 18 19:54:18.458170 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:18.458136    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:23.242786 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:23.242741    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:54:23.462232 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:23.462205    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:54:23.462705 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:23.462677    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:24.361785 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:24.361756    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:33.242132 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:33.242084    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:54:33.462650 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:33.462601    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:34.361238 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:34.361201    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:43.242112 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:43.242061    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:54:43.463522 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:43.463487    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:44.362041 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:44.362004    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:54:53.242479 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:53.242447    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:54:53.463094 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:53.463057    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:54:54.361786 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:54:54.361752    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:55:01.479676 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:01.479643    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:55:01.480177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:01.479901    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" containerID="cri-o://8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698" gracePeriod=30
Sep 18 19:55:01.480177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:01.479944    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kube-rbac-proxy" containerID="cri-o://c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1" gracePeriod=30
Sep 18 19:55:01.662965 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:01.662925    2567 generic.go:356] "Generic (PLEG): container finished" podID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerID="c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1" exitCode=2
Sep 18 19:55:01.663115 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:01.662990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerDied","Data":"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"}
Sep 18 19:55:03.237900 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:03.237853    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 18 19:55:03.242160 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:03.242127    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 18 19:55:03.463730 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:03.463695    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:55:04.361434 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:04.361396    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:55:06.324830 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.324804    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:55:06.375115 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375087    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config\") pod \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") "
Sep 18 19:55:06.375291 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375130    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2qlx\" (UniqueName: \"kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx\") pod \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") "
Sep 18 19:55:06.375291 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375157    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") pod \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") "
Sep 18 19:55:06.375291 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375213    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert\") pod \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") "
Sep 18 19:55:06.375291 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375276    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location\") pod \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\" (UID: \"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54\") "
Sep 18 19:55:06.375635 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375592    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:06.375763 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375637    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:06.375763 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.375668    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:06.377331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.377300    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx" (OuterVolumeSpecName: "kube-api-access-s2qlx") pod "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54"). InnerVolumeSpecName "kube-api-access-s2qlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:06.377331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.377317    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" (UID: "310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:06.476706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.476621    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:06.476706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.476649    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:06.476706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.476661    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:06.476706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.476672    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qlx\" (UniqueName: \"kubernetes.io/projected/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-kube-api-access-s2qlx\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:06.476706 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.476681    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:06.692403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.687715    2567 generic.go:356] "Generic (PLEG): container finished" podID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerID="8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698" exitCode=0
Sep 18 19:55:06.692403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.687834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerDied","Data":"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"}
Sep 18 19:55:06.692403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.687956    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz" event={"ID":"310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54","Type":"ContainerDied","Data":"071bcb9231d6033a0d789aad409b92f41c64f2d0f9bfe36c0cd57185d38e1654"}
Sep 18 19:55:06.692403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.688008    2567 scope.go:117] "RemoveContainer" containerID="c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"
Sep 18 19:55:06.692403 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.688245    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"
Sep 18 19:55:06.703193 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.703173    2567 scope.go:117] "RemoveContainer" containerID="8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"
Sep 18 19:55:06.711130 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.711111    2567 scope.go:117] "RemoveContainer" containerID="fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"
Sep 18 19:55:06.713550 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.713528    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:55:06.717474 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.717456    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-mhxmz"]
Sep 18 19:55:06.720120 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.720105    2567 scope.go:117] "RemoveContainer" containerID="c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"
Sep 18 19:55:06.720451 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:06.720375    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1\": container with ID starting with c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1 not found: ID does not exist" containerID="c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"
Sep 18 19:55:06.720451 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.720411    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1"} err="failed to get container status \"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1\": rpc error: code = NotFound desc = could not find container \"c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1\": container with ID starting with c9a03fed0f700ba2504fd8e81d45ff9d15a3aca1c8b1ff4d36d2d213447386a1 not found: ID does not exist"
Sep 18 19:55:06.720451 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.720434    2567 scope.go:117] "RemoveContainer" containerID="8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"
Sep 18 19:55:06.720698 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:06.720680    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698\": container with ID starting with 8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698 not found: ID does not exist" containerID="8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"
Sep 18 19:55:06.720763 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.720707    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698"} err="failed to get container status \"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698\": rpc error: code = NotFound desc = could not find container \"8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698\": container with ID starting with 8b39dfe7ce0e641528cccd1ee92a8e0d811664538964e30b2c7c2b568d5b2698 not found: ID does not exist"
Sep 18 19:55:06.720763 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.720730    2567 scope.go:117] "RemoveContainer" containerID="fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"
Sep 18 19:55:06.721028 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:06.721011    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2\": container with ID starting with fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2 not found: ID does not exist" containerID="fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"
Sep 18 19:55:06.721070 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:06.721033    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2"} err="failed to get container status \"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2\": rpc error: code = NotFound desc = could not find container \"fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2\": container with ID starting with fa64e6697e569dc3040de37df387269ccddf4e8171457c68512c590feb817dd2 not found: ID does not exist"
Sep 18 19:55:08.352906 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:08.352876    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" path="/var/lib/kubelet/pods/310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54/volumes"
Sep 18 19:55:13.463528 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:13.463487    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:55:14.362449 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:14.362421    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:55:23.463540 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:23.463507    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:55:41.639779 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.639741    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:55:41.640303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.640016    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" containerID="cri-o://45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3" gracePeriod=30
Sep 18 19:55:41.640303 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.640045    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kube-rbac-proxy" containerID="cri-o://1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087" gracePeriod=30
Sep 18 19:55:41.782189 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.782159    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:55:41.782471 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.782438    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kube-rbac-proxy" containerID="cri-o://abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d" gracePeriod=30
Sep 18 19:55:41.782614 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.782426    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" containerID="cri-o://c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a" gracePeriod=30
Sep 18 19:55:41.854907 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.854872    2567 generic.go:356] "Generic (PLEG): container finished" podID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerID="1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087" exitCode=2
Sep 18 19:55:41.855075 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:41.854940    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerDied","Data":"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"}
Sep 18 19:55:42.861949 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:42.861915    2567 generic.go:356] "Generic (PLEG): container finished" podID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerID="abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d" exitCode=2
Sep 18 19:55:42.862343 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:42.861964    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerDied","Data":"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"}
Sep 18 19:55:43.458825 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:43.458781    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 18 19:55:43.463133 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:43.463107    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 18 19:55:44.356267 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:44.356219    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 18 19:55:44.361949 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:44.361920    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 18 19:55:46.119816 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.119787    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:55:46.244889 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.244795    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location\") pod \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") "
Sep 18 19:55:46.244889 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.244850    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert\") pod \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") "
Sep 18 19:55:46.245116 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.244908    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77vz5\" (UniqueName: \"kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5\") pod \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") "
Sep 18 19:55:46.245116 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.244949    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config\") pod \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") "
Sep 18 19:55:46.245116 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.244993    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") pod \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\" (UID: \"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd\") "
Sep 18 19:55:46.245244 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.245107    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:46.245293 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.245255    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:46.245293 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.245277    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.245394 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.245305    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:46.247041 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.247018    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5" (OuterVolumeSpecName: "kube-api-access-77vz5") pod "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd"). InnerVolumeSpecName "kube-api-access-77vz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:46.247114 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.247017    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" (UID: "faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:46.346578 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.346544    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.346578 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.346575    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vz5\" (UniqueName: \"kubernetes.io/projected/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-api-access-77vz5\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.346578 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.346586    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.346818 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.346596    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.589152 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.589129    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:55:46.648944 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.648916    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config\") pod \"f0bcc744-7e00-4299-950c-59018f18ecc6\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") "
Sep 18 19:55:46.648944 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.648952    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t5wm\" (UniqueName: \"kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm\") pod \"f0bcc744-7e00-4299-950c-59018f18ecc6\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") "
Sep 18 19:55:46.649157 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.648985    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location\") pod \"f0bcc744-7e00-4299-950c-59018f18ecc6\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") "
Sep 18 19:55:46.649157 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.649004    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert\") pod \"f0bcc744-7e00-4299-950c-59018f18ecc6\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") "
Sep 18 19:55:46.649157 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.649078    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls\") pod \"f0bcc744-7e00-4299-950c-59018f18ecc6\" (UID: \"f0bcc744-7e00-4299-950c-59018f18ecc6\") "
Sep 18 19:55:46.649390 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.649341    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0bcc744-7e00-4299-950c-59018f18ecc6" (UID: "f0bcc744-7e00-4299-950c-59018f18ecc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:46.649476 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.649342    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0bcc744-7e00-4299-950c-59018f18ecc6" (UID: "f0bcc744-7e00-4299-950c-59018f18ecc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:46.649476 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.649404    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0bcc744-7e00-4299-950c-59018f18ecc6" (UID: "f0bcc744-7e00-4299-950c-59018f18ecc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:46.651111 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.651086    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0bcc744-7e00-4299-950c-59018f18ecc6" (UID: "f0bcc744-7e00-4299-950c-59018f18ecc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:46.651195 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.651115    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm" (OuterVolumeSpecName: "kube-api-access-9t5wm") pod "f0bcc744-7e00-4299-950c-59018f18ecc6" (UID: "f0bcc744-7e00-4299-950c-59018f18ecc6"). InnerVolumeSpecName "kube-api-access-9t5wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:46.750243 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.750206    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bcc744-7e00-4299-950c-59018f18ecc6-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.750243 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.750240    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.750243 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.750249    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bcc744-7e00-4299-950c-59018f18ecc6-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.750521 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.750258    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.750521 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.750269    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5wm\" (UniqueName: \"kubernetes.io/projected/f0bcc744-7e00-4299-950c-59018f18ecc6-kube-api-access-9t5wm\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:55:46.883614 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.883580    2567 generic.go:356] "Generic (PLEG): container finished" podID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerID="c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a" exitCode=0
Sep 18 19:55:46.883811 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.883638    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"
Sep 18 19:55:46.883811 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.883666    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerDied","Data":"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"}
Sep 18 19:55:46.883811 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.883730    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl" event={"ID":"faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd","Type":"ContainerDied","Data":"f7ab76248102806c00d5bd5a3f10a0232ec0dcca42142bef7c1f0ffed652dc23"}
Sep 18 19:55:46.883811 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.883766    2567 scope.go:117] "RemoveContainer" containerID="abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"
Sep 18 19:55:46.885860 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.885836    2567 generic.go:356] "Generic (PLEG): container finished" podID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerID="45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3" exitCode=0
Sep 18 19:55:46.885974 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.885870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerDied","Data":"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"}
Sep 18 19:55:46.885974 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.885922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf" event={"ID":"f0bcc744-7e00-4299-950c-59018f18ecc6","Type":"ContainerDied","Data":"8f136be2c4d2ca2b14b3195f96567db2e8cbd047fbb4227dbab3f300ecbdd06b"}
Sep 18 19:55:46.886087 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.886007    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"
Sep 18 19:55:46.893585 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.893567    2567 scope.go:117] "RemoveContainer" containerID="c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"
Sep 18 19:55:46.900270 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.900245    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:55:46.901809 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.901786    2567 scope.go:117] "RemoveContainer" containerID="7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"
Sep 18 19:55:46.903881 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.903858    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e0026-predictor-85549564df-cq4rl"]
Sep 18 19:55:46.910134 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910120    2567 scope.go:117] "RemoveContainer" containerID="abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"
Sep 18 19:55:46.910442 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.910425    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d\": container with ID starting with abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d not found: ID does not exist" containerID="abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"
Sep 18 19:55:46.910498 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910450    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d"} err="failed to get container status \"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d\": rpc error: code = NotFound desc = could not find container \"abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d\": container with ID starting with abae9fe4d54ae9a96824891f5ad221768cf5940847151d8890ae2a524210c91d not found: ID does not exist"
Sep 18 19:55:46.910498 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910467    2567 scope.go:117] "RemoveContainer" containerID="c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"
Sep 18 19:55:46.910708 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.910693    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a\": container with ID starting with c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a not found: ID does not exist" containerID="c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"
Sep 18 19:55:46.910747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910713    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a"} err="failed to get container status \"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a\": rpc error: code = NotFound desc = could not find container \"c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a\": container with ID starting with c1036dea4fa866738982e62b3461bafc36e9bf47c4355dca668f9e64ec96457a not found: ID does not exist"
Sep 18 19:55:46.910747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910728    2567 scope.go:117] "RemoveContainer" containerID="7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"
Sep 18 19:55:46.910939 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.910922    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154\": container with ID starting with 7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154 not found: ID does not exist" containerID="7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"
Sep 18 19:55:46.910980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910942    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154"} err="failed to get container status \"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154\": rpc error: code = NotFound desc = could not find container \"7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154\": container with ID starting with 7865c8d75fbc77112fe49d123099254323d6eb475fe678bacdfeb0493c050154 not found: ID does not exist"
Sep 18 19:55:46.910980 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.910954    2567 scope.go:117] "RemoveContainer" containerID="1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"
Sep 18 19:55:46.919871 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.919853    2567 scope.go:117] "RemoveContainer" containerID="45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"
Sep 18 19:55:46.919935 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.919884    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:55:46.924794 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.924771    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e0026-predictor-5f4bf59955-jtnrf"]
Sep 18 19:55:46.928003 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.927985    2567 scope.go:117] "RemoveContainer" containerID="a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"
Sep 18 19:55:46.935519 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.935501    2567 scope.go:117] "RemoveContainer" containerID="1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"
Sep 18 19:55:46.935786 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.935768    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087\": container with ID starting with 1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087 not found: ID does not exist" containerID="1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"
Sep 18 19:55:46.935835 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.935794    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087"} err="failed to get container status \"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087\": rpc error: code = NotFound desc = could not find container \"1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087\": container with ID starting with 1f826272081909d820b5cc2f20f354732f2890d717031e25618603b08c0ea087 not found: ID does not exist"
Sep 18 19:55:46.935835 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.935813    2567 scope.go:117] "RemoveContainer" containerID="45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"
Sep 18 19:55:46.936102 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.936086    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3\": container with ID starting with 45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3 not found: ID does not exist" containerID="45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"
Sep 18 19:55:46.936139 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.936109    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3"} err="failed to get container status \"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3\": rpc error: code = NotFound desc = could not find container \"45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3\": container with ID starting with 45bf26d0e6ae4f06ca97eacf588d8695cb997b42c70c486ad34a8c4ae4e015b3 not found: ID does not exist"
Sep 18 19:55:46.936139 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.936126    2567 scope.go:117] "RemoveContainer" containerID="a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"
Sep 18 19:55:46.936390 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:55:46.936356    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff\": container with ID starting with a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff not found: ID does not exist" containerID="a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"
Sep 18 19:55:46.936434 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:46.936393    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff"} err="failed to get container status \"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff\": rpc error: code = NotFound desc = could not find container \"a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff\": container with ID starting with a266959e301c8aa41578d5cae4c99dc9ad630e6515f10457009381141e53c2ff not found: ID does not exist"
Sep 18 19:55:48.353994 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:48.353958    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" path="/var/lib/kubelet/pods/f0bcc744-7e00-4299-950c-59018f18ecc6/volumes"
Sep 18 19:55:48.354690 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:48.354671    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" path="/var/lib/kubelet/pods/faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd/volumes"
Sep 18 19:55:51.772438 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772401    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772733    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772744    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772757    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772762    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772773    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772779    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772790    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772795    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="storage-initializer"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772802    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kube-rbac-proxy"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772808    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kube-rbac-proxy"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772816    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kube-rbac-proxy"
Sep 18 19:55:51.772814 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772821    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kube-rbac-proxy"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772827    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772832    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772838    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772843    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772854    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kube-rbac-proxy"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772859    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kube-rbac-proxy"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772908    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772915    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772922    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kube-rbac-proxy"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772928    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bcc744-7e00-4299-950c-59018f18ecc6" containerName="kube-rbac-proxy"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772934    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="310f5d1e-6c8b-47c0-aaf2-b8b04c6eed54" containerName="kserve-container"
Sep 18 19:55:51.773177 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.772939    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf2e9f7-e654-4e5b-89e1-ddb348f7d0bd" containerName="kube-rbac-proxy"
Sep 18 19:55:51.776679 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.776661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.778741 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.778715    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-91b5b-kube-rbac-proxy-sar-config\""
Sep 18 19:55:51.778854 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.778742    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:55:51.778854 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.778716    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:55:51.779186 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.779165    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-91b5b-predictor-serving-cert\""
Sep 18 19:55:51.779186 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.779178    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:55:51.779331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.779183    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 19:55:51.779331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.779166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:55:51.786869 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.786847    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:55:51.895968 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.895917    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.895968 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.895972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.896200 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.896033    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7hb\" (UniqueName: \"kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.896200 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.896056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.896200 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.896089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.996843 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.996807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7hb\" (UniqueName: \"kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.996843 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.996849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997111 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.996948    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997111 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.996988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997111 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.997007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997457 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.997433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997683 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.997663    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.997739 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.997720    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:51.999340 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:51.999321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:52.004764 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:52.004736    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7hb\" (UniqueName: \"kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb\") pod \"isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") " pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:52.088530 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:52.088471    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:52.364709 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:52.364683    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:55:52.367187 ip-10-0-143-198 kubenswrapper[2567]: W0918 19:55:52.367160    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15831d0b_ce18_4541_8c7d_d2995e430b89.slice/crio-bc01e9c51406b65700188955bedfb2bcb4287f4796d5c62618af31097f1d5fe3 WatchSource:0}: Error finding container bc01e9c51406b65700188955bedfb2bcb4287f4796d5c62618af31097f1d5fe3: Status 404 returned error can't find the container with id bc01e9c51406b65700188955bedfb2bcb4287f4796d5c62618af31097f1d5fe3
Sep 18 19:55:52.917064 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:52.917029    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerStarted","Data":"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"}
Sep 18 19:55:52.917464 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:52.917098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerStarted","Data":"bc01e9c51406b65700188955bedfb2bcb4287f4796d5c62618af31097f1d5fe3"}
Sep 18 19:55:56.939534 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:56.939498    2567 generic.go:356] "Generic (PLEG): container finished" podID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerID="d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405" exitCode=0
Sep 18 19:55:56.939997 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:56.939580    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerDied","Data":"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"}
Sep 18 19:55:57.946751 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.946714    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerStarted","Data":"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"}
Sep 18 19:55:57.947131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.946790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerStarted","Data":"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"}
Sep 18 19:55:57.947131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.946817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerStarted","Data":"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"}
Sep 18 19:55:57.947131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.947038    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:57.947131 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.947091    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:57.948468 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.948445    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:55:57.969932 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:57.969882    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podStartSLOduration=3.138236959 podStartE2EDuration="6.969865736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831628777s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:57.967353934 +0000 UTC m=+1055.746724516" watchObservedRunningTime="2026-09-18 19:55:57.969865736 +0000 UTC m=+1055.749236317"
Sep 18 19:55:58.951066 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:58.951036    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:55:58.951478 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:58.951208    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:55:58.952314 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:58.952289    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:55:59.956273 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:59.956225    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:55:59.956723 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:55:59.956703    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:04.960788 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:04.960760    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:56:04.961430 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:04.961397    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:04.961708 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:04.961674    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:14.962279 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:14.962172    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:14.962792 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:14.962658    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:24.961918 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:24.961876    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:24.962397 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:24.962259    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:34.961853 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:34.961804    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:34.962392 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:34.962352    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:44.961856 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:44.961812    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:44.962320 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:44.962295    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:56:54.961680 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:54.961626    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:56:54.962199 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:56:54.962134    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:57:04.961568 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:04.961517    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:57:04.961999 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:04.961936    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:57:14.962512 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:14.962479    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:14.962948 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:14.962593    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:26.923562 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:26.923523    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:57:26.924133 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:26.923796    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" containerID="cri-o://fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629" gracePeriod=30
Sep 18 19:57:26.924133 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:26.923826    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" containerID="cri-o://f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01" gracePeriod=30
Sep 18 19:57:26.924133 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:26.923826    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" containerID="cri-o://30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28" gracePeriod=30
Sep 18 19:57:27.333566 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:27.333529    2567 generic.go:356] "Generic (PLEG): container finished" podID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerID="30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28" exitCode=2
Sep 18 19:57:27.333747 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:27.333593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerDied","Data":"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"}
Sep 18 19:57:29.956691 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:29.956649    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:32.359862 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:32.359816    2567 generic.go:356] "Generic (PLEG): container finished" podID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerID="fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629" exitCode=0
Sep 18 19:57:32.360236 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:32.359884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerDied","Data":"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"}
Sep 18 19:57:34.956727 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:34.956684    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:34.962237 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:34.962197    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:57:34.962633 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:34.962607    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:57:39.956537 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:39.956493    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:39.956926 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:39.956627    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:44.956763 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:44.956723    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:44.962331 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:44.962293    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:57:44.962671 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:44.962643    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:57:49.956684 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:49.956642    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:54.957232 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:54.957189    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 18 19:57:54.961585 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:54.961549    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 18 19:57:54.961740 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:54.961684    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:54.961927 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:54.961901    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:57:54.962021 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:54.962009    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:57.065132 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.065110    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:57.127343 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127319    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp7hb\" (UniqueName: \"kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb\") pod \"15831d0b-ce18-4541-8c7d-d2995e430b89\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") "
Sep 18 19:57:57.127489 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127355    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config\") pod \"15831d0b-ce18-4541-8c7d-d2995e430b89\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") "
Sep 18 19:57:57.127489 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127443    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location\") pod \"15831d0b-ce18-4541-8c7d-d2995e430b89\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") "
Sep 18 19:57:57.127580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127494    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert\") pod \"15831d0b-ce18-4541-8c7d-d2995e430b89\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") "
Sep 18 19:57:57.127580 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127528    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls\") pod \"15831d0b-ce18-4541-8c7d-d2995e430b89\" (UID: \"15831d0b-ce18-4541-8c7d-d2995e430b89\") "
Sep 18 19:57:57.127794 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127765    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15831d0b-ce18-4541-8c7d-d2995e430b89" (UID: "15831d0b-ce18-4541-8c7d-d2995e430b89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:57.127794 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127777    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15831d0b-ce18-4541-8c7d-d2995e430b89" (UID: "15831d0b-ce18-4541-8c7d-d2995e430b89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:57.127935 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.127857    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15831d0b-ce18-4541-8c7d-d2995e430b89" (UID: "15831d0b-ce18-4541-8c7d-d2995e430b89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:57.129443 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.129422    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb" (OuterVolumeSpecName: "kube-api-access-mp7hb") pod "15831d0b-ce18-4541-8c7d-d2995e430b89" (UID: "15831d0b-ce18-4541-8c7d-d2995e430b89"). InnerVolumeSpecName "kube-api-access-mp7hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:57.129545 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.129527    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15831d0b-ce18-4541-8c7d-d2995e430b89" (UID: "15831d0b-ce18-4541-8c7d-d2995e430b89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:57.228581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.228521    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15831d0b-ce18-4541-8c7d-d2995e430b89-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:57:57.228581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.228544    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:57:57.228581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.228554    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15831d0b-ce18-4541-8c7d-d2995e430b89-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:57:57.228581 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.228579    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7hb\" (UniqueName: \"kubernetes.io/projected/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-api-access-mp7hb\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:57:57.228784 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.228588    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15831d0b-ce18-4541-8c7d-d2995e430b89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 19:57:57.469638 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.469606    2567 generic.go:356] "Generic (PLEG): container finished" podID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerID="f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01" exitCode=0
Sep 18 19:57:57.469787 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.469656    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerDied","Data":"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"}
Sep 18 19:57:57.469787 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.469698    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"
Sep 18 19:57:57.469787 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.469712    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs" event={"ID":"15831d0b-ce18-4541-8c7d-d2995e430b89","Type":"ContainerDied","Data":"bc01e9c51406b65700188955bedfb2bcb4287f4796d5c62618af31097f1d5fe3"}
Sep 18 19:57:57.469787 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.469741    2567 scope.go:117] "RemoveContainer" containerID="f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"
Sep 18 19:57:57.477844 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.477830    2567 scope.go:117] "RemoveContainer" containerID="30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"
Sep 18 19:57:57.484724 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.484704    2567 scope.go:117] "RemoveContainer" containerID="fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"
Sep 18 19:57:57.490045 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.490020    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:57:57.492361 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.492341    2567 scope.go:117] "RemoveContainer" containerID="d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"
Sep 18 19:57:57.493471 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.493451    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-91b5b-predictor-578588fc8f-rnhhs"]
Sep 18 19:57:57.499449 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.499431    2567 scope.go:117] "RemoveContainer" containerID="f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"
Sep 18 19:57:57.499683 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:57:57.499663    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01\": container with ID starting with f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01 not found: ID does not exist" containerID="f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"
Sep 18 19:57:57.499735 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.499691    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01"} err="failed to get container status \"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01\": rpc error: code = NotFound desc = could not find container \"f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01\": container with ID starting with f06479b20989a706dc64bcaa717734355dd4d1d4e6b40135357d8b8f9465ef01 not found: ID does not exist"
Sep 18 19:57:57.499735 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.499709    2567 scope.go:117] "RemoveContainer" containerID="30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"
Sep 18 19:57:57.499927 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:57:57.499913    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28\": container with ID starting with 30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28 not found: ID does not exist" containerID="30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"
Sep 18 19:57:57.499974 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.499932    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28"} err="failed to get container status \"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28\": rpc error: code = NotFound desc = could not find container \"30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28\": container with ID starting with 30fe480d037f10094979d59640e872b0db7ae434853baf6b34780ef32ad7ab28 not found: ID does not exist"
Sep 18 19:57:57.499974 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.499953    2567 scope.go:117] "RemoveContainer" containerID="fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"
Sep 18 19:57:57.500152 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:57:57.500134    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629\": container with ID starting with fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629 not found: ID does not exist" containerID="fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"
Sep 18 19:57:57.500196 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.500158    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629"} err="failed to get container status \"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629\": rpc error: code = NotFound desc = could not find container \"fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629\": container with ID starting with fc3c2abb9536ea2299696590d802784506e264e2d05e33b0055c759ad08d5629 not found: ID does not exist"
Sep 18 19:57:57.500196 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.500174    2567 scope.go:117] "RemoveContainer" containerID="d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"
Sep 18 19:57:57.500432 ip-10-0-143-198 kubenswrapper[2567]: E0918 19:57:57.500416    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405\": container with ID starting with d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405 not found: ID does not exist" containerID="d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"
Sep 18 19:57:57.500477 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:57.500435    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405"} err="failed to get container status \"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405\": rpc error: code = NotFound desc = could not find container \"d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405\": container with ID starting with d8934db4a293fd6ee45b37f008b8178c62be9279b8ef2116a38f04accf044405 not found: ID does not exist"
Sep 18 19:57:58.353485 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:57:58.353456    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" path="/var/lib/kubelet/pods/15831d0b-ce18-4541-8c7d-d2995e430b89/volumes"
Sep 18 19:58:24.310723 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:58:24.310694    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 19:58:24.313461 ip-10-0-143-198 kubenswrapper[2567]: I0918 19:58:24.313440    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 20:00:12.354573 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354540    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354844    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="storage-initializer"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354854    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="storage-initializer"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354864    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354870    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354880    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354885    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354903    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354908    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354963    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kserve-container"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354972    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="agent"
Sep 18 20:00:12.355174 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.354979    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="15831d0b-ce18-4541-8c7d-d2995e430b89" containerName="kube-rbac-proxy"
Sep 18 20:00:12.358307 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.358290    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.360559 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.360532    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:00:12.360703 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.360537    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:00:12.360703 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.360596    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:00:12.360807 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.360723    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:00:12.360885 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.360872    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:00:12.361019 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.361005    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:00:12.361090 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.361076    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 20:00:12.367642 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.367617    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:00:12.449123 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.449086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.449350 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.449144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.449350 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.449241    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.449350 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.449278    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qct\" (UniqueName: \"kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.449513 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.449358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.550854 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.550807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.550854 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.550856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qct\" (UniqueName: \"kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551133 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.550945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551133 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:00:12.550974    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:00:12.551133 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.551031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551133 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:00:12.551058    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls podName:d50bd5e5-8231-46d5-b611-a10f73987fca nodeName:}" failed. No retries permitted until 2026-09-18 20:00:13.051038355 +0000 UTC m=+1310.830408918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls") pod "isvc-canary-create-predictor-68984b9498-rd2x9" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:00:12.551133 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.551111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551531 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.551506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551755 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.551739    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.551810 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.551774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:12.561265 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:12.561236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qct\" (UniqueName: \"kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:13.057189 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:13.057152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:13.059695 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:13.059667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-rd2x9\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:13.270676 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:13.270630    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:13.539888 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:13.539861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:00:13.542399 ip-10-0-143-198 kubenswrapper[2567]: W0918 20:00:13.542346    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50bd5e5_8231_46d5_b611_a10f73987fca.slice/crio-c17eac0181d0d7d582a1cfab829dbdf994fe5bd8d714b297c12bfb183d349c44 WatchSource:0}: Error finding container c17eac0181d0d7d582a1cfab829dbdf994fe5bd8d714b297c12bfb183d349c44: Status 404 returned error can't find the container with id c17eac0181d0d7d582a1cfab829dbdf994fe5bd8d714b297c12bfb183d349c44
Sep 18 20:00:14.057206 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:14.057166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerStarted","Data":"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"}
Sep 18 20:00:14.057397 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:14.057240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerStarted","Data":"c17eac0181d0d7d582a1cfab829dbdf994fe5bd8d714b297c12bfb183d349c44"}
Sep 18 20:00:18.074851 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:18.074822    2567 generic.go:356] "Generic (PLEG): container finished" podID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerID="85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba" exitCode=0
Sep 18 20:00:18.075221 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:18.074867    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerDied","Data":"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"}
Sep 18 20:00:19.081337 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.081303    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerStarted","Data":"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"}
Sep 18 20:00:19.081759 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.081389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerStarted","Data":"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"}
Sep 18 20:00:19.081759 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.081679    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:19.081880 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.081837    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:19.083062 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.083034    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:19.098574 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:19.098537    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podStartSLOduration=3.096897308 podStartE2EDuration="7.098524489s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.001627181s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:19.096811618 +0000 UTC m=+1316.876182200" watchObservedRunningTime="2026-09-18 20:00:19.098524489 +0000 UTC m=+1316.877895072"
Sep 18 20:00:20.085543 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:20.085506    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:21.090207 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:21.090170    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:26.094900 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:26.094870    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:00:26.095359 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:26.095333    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:36.095697 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:36.095658    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:46.095623 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:46.095583    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:00:56.096151 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:00:56.096106    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:01:06.095438 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:06.095395    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:01:16.096046 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:16.095998    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:01:26.096057 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:26.096017    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 18 20:01:35.624716 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:35.624680    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:01:35.625324 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:35.624983    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container" containerID="cri-o://0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767" gracePeriod=30
Sep 18 20:01:35.625324 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:35.625029    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kube-rbac-proxy" containerID="cri-o://a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7" gracePeriod=30
Sep 18 20:01:35.625807 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:35.625508    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:01:36.090567 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:36.090524    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 18 20:01:36.406139 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:36.406054    2567 generic.go:356] "Generic (PLEG): container finished" podID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerID="a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7" exitCode=2
Sep 18 20:01:36.406302 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:36.406131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerDied","Data":"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"}
Sep 18 20:01:40.671685 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.671660    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:01:40.720225 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720143    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert\") pod \"d50bd5e5-8231-46d5-b611-a10f73987fca\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") "
Sep 18 20:01:40.720225 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720191    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config\") pod \"d50bd5e5-8231-46d5-b611-a10f73987fca\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") "
Sep 18 20:01:40.720225 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720214    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") pod \"d50bd5e5-8231-46d5-b611-a10f73987fca\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") "
Sep 18 20:01:40.720535 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720248    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99qct\" (UniqueName: \"kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct\") pod \"d50bd5e5-8231-46d5-b611-a10f73987fca\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") "
Sep 18 20:01:40.720535 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720286    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location\") pod \"d50bd5e5-8231-46d5-b611-a10f73987fca\" (UID: \"d50bd5e5-8231-46d5-b611-a10f73987fca\") "
Sep 18 20:01:40.720647 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720607    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d50bd5e5-8231-46d5-b611-a10f73987fca" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:40.720704 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720634    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d50bd5e5-8231-46d5-b611-a10f73987fca" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:40.720704 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.720653    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d50bd5e5-8231-46d5-b611-a10f73987fca" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:40.722390 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.722358    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d50bd5e5-8231-46d5-b611-a10f73987fca" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:40.722500 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.722439    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct" (OuterVolumeSpecName: "kube-api-access-99qct") pod "d50bd5e5-8231-46d5-b611-a10f73987fca" (UID: "d50bd5e5-8231-46d5-b611-a10f73987fca"). InnerVolumeSpecName "kube-api-access-99qct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:40.821172 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.821131    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d50bd5e5-8231-46d5-b611-a10f73987fca-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:01:40.821172 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.821162    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qct\" (UniqueName: \"kubernetes.io/projected/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-api-access-99qct\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:01:40.821172 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.821173    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d50bd5e5-8231-46d5-b611-a10f73987fca-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:01:40.821172 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.821183    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:01:40.821556 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:40.821193    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d50bd5e5-8231-46d5-b611-a10f73987fca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:01:41.428711 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.428673    2567 generic.go:356] "Generic (PLEG): container finished" podID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerID="0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767" exitCode=0
Sep 18 20:01:41.428926 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.428750    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"
Sep 18 20:01:41.428926 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.428749    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerDied","Data":"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"}
Sep 18 20:01:41.428926 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.428884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9" event={"ID":"d50bd5e5-8231-46d5-b611-a10f73987fca","Type":"ContainerDied","Data":"c17eac0181d0d7d582a1cfab829dbdf994fe5bd8d714b297c12bfb183d349c44"}
Sep 18 20:01:41.428926 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.428910    2567 scope.go:117] "RemoveContainer" containerID="a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"
Sep 18 20:01:41.438534 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.438517    2567 scope.go:117] "RemoveContainer" containerID="0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"
Sep 18 20:01:41.446358 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.446343    2567 scope.go:117] "RemoveContainer" containerID="85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"
Sep 18 20:01:41.450663 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.450643    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:01:41.455420 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.455336    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-rd2x9"]
Sep 18 20:01:41.455498 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.455435    2567 scope.go:117] "RemoveContainer" containerID="a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"
Sep 18 20:01:41.455724 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:01:41.455703    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7\": container with ID starting with a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7 not found: ID does not exist" containerID="a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"
Sep 18 20:01:41.455806 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.455736    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7"} err="failed to get container status \"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7\": rpc error: code = NotFound desc = could not find container \"a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7\": container with ID starting with a7d5ecc9a70dcafeb0b2d6c53ef5097c0626eed4c4eb270bc23bb9b9de6331d7 not found: ID does not exist"
Sep 18 20:01:41.455806 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.455760    2567 scope.go:117] "RemoveContainer" containerID="0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"
Sep 18 20:01:41.456057 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:01:41.456039    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767\": container with ID starting with 0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767 not found: ID does not exist" containerID="0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"
Sep 18 20:01:41.456096 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.456064    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767"} err="failed to get container status \"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767\": rpc error: code = NotFound desc = could not find container \"0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767\": container with ID starting with 0afd7661cb7adbf2ebad91d8f708eec7deda22f7b5506b830142d5ede7b67767 not found: ID does not exist"
Sep 18 20:01:41.456096 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.456081    2567 scope.go:117] "RemoveContainer" containerID="85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"
Sep 18 20:01:41.456325 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:01:41.456307    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba\": container with ID starting with 85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba not found: ID does not exist" containerID="85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"
Sep 18 20:01:41.456447 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:41.456331    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba"} err="failed to get container status \"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba\": rpc error: code = NotFound desc = could not find container \"85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba\": container with ID starting with 85b0e37e38396a9c2183ec208b5c7b6bbee637576dd2089aa288ae523dcfdfba not found: ID does not exist"
Sep 18 20:01:42.353090 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:42.353058    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" path="/var/lib/kubelet/pods/d50bd5e5-8231-46d5-b611-a10f73987fca/volumes"
Sep 18 20:01:47.626341 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626304    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626680    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kube-rbac-proxy"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626693    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kube-rbac-proxy"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626755    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="storage-initializer"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626765    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="storage-initializer"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626774    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626780    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626846    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kserve-container"
Sep 18 20:01:47.626884 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.626855    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bd5e5-8231-46d5-b611-a10f73987fca" containerName="kube-rbac-proxy"
Sep 18 20:01:47.630120 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.630096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.632437 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.632415    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:01:47.632761 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.632704    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:01:47.632761 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.632727    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:01:47.633220 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.633139    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:01:47.633220 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.633161    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:01:47.633220 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.633171    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:01:47.633220 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.633177    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 20:01:47.641209 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.641188    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:01:47.680219 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.680192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.680409 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.680231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.680409 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.680270    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csq8w\" (UniqueName: \"kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.680409 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.680296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.680409 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.680314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781252 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781215    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781252 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781265    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781633 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781633 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781633 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csq8w\" (UniqueName: \"kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.781633 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:01:47.781495    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 20:01:47.781633 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:01:47.781575    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls podName:45cdd521-d5fa-431d-9d4e-20c68e5cfd9a nodeName:}" failed. No retries permitted until 2026-09-18 20:01:48.28155474 +0000 UTC m=+1406.060925314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls") pod "isvc-canary-promote-predictor-cb75746f8-zgvp4" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 18 20:01:47.781869 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.781638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.782108 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.782086    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.782145 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.782086    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:47.790199 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:47.790168    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csq8w\" (UniqueName: \"kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:48.285773 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:48.285721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:48.288168 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:48.288137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-zgvp4\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:48.541667 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:48.541566    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:48.812394 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:48.812357    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:01:48.814813 ip-10-0-143-198 kubenswrapper[2567]: W0918 20:01:48.814786    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cdd521_d5fa_431d_9d4e_20c68e5cfd9a.slice/crio-d126a42210f6bce3ca5c8413ef621e489e2089f3072d1dc40e1c63c2bf964e2c WatchSource:0}: Error finding container d126a42210f6bce3ca5c8413ef621e489e2089f3072d1dc40e1c63c2bf964e2c: Status 404 returned error can't find the container with id d126a42210f6bce3ca5c8413ef621e489e2089f3072d1dc40e1c63c2bf964e2c
Sep 18 20:01:49.470971 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:49.470936    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerStarted","Data":"66f75168a02591effb8ca63a0a862dc5383ce4da16927867e32dd9af7de550bd"}
Sep 18 20:01:49.471203 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:49.470992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerStarted","Data":"d126a42210f6bce3ca5c8413ef621e489e2089f3072d1dc40e1c63c2bf964e2c"}
Sep 18 20:01:53.490092 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:53.490060    2567 generic.go:356] "Generic (PLEG): container finished" podID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerID="66f75168a02591effb8ca63a0a862dc5383ce4da16927867e32dd9af7de550bd" exitCode=0
Sep 18 20:01:53.490586 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:53.490144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerDied","Data":"66f75168a02591effb8ca63a0a862dc5383ce4da16927867e32dd9af7de550bd"}
Sep 18 20:01:54.496874 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.496837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerStarted","Data":"c73d56d28af6cb44901d9e0f83b37f9e6e83510660a3ec506a7aadba9ed2f440"}
Sep 18 20:01:54.497303 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.496898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerStarted","Data":"a685fcde488a33378607ba0432ca4e24bcd2bf4612590308fff11d8a1820fc4c"}
Sep 18 20:01:54.497303 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.497161    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:54.497433 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.497339    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:01:54.498729 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.498703    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:01:54.515316 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:54.515242    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podStartSLOduration=3.487967286 podStartE2EDuration="7.515229721s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.027262435s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:54.51475237 +0000 UTC m=+1412.294122951" watchObservedRunningTime="2026-09-18 20:01:54.515229721 +0000 UTC m=+1412.294600327"
Sep 18 20:01:55.501796 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:01:55.501756    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:00.506381 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:00.506342    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:02:00.506943 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:00.506908    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:10.507686 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:10.507601    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:20.506851 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:20.506810    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:30.507824 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:30.507786    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:40.507590 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:40.507547    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:02:50.507289 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:02:50.507246    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:03:00.507481 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:00.507439    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:03:10.507506 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:10.507477    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:03:17.740959 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.740930    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:03:17.741429 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.741162    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" containerID="cri-o://a685fcde488a33378607ba0432ca4e24bcd2bf4612590308fff11d8a1820fc4c" gracePeriod=30
Sep 18 20:03:17.741429 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.741202    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kube-rbac-proxy" containerID="cri-o://c73d56d28af6cb44901d9e0f83b37f9e6e83510660a3ec506a7aadba9ed2f440" gracePeriod=30
Sep 18 20:03:17.860999 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.860976    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:03:17.862107 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.862083    2567 generic.go:356] "Generic (PLEG): container finished" podID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerID="c73d56d28af6cb44901d9e0f83b37f9e6e83510660a3ec506a7aadba9ed2f440" exitCode=2
Sep 18 20:03:17.864740 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.864717    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerDied","Data":"c73d56d28af6cb44901d9e0f83b37f9e6e83510660a3ec506a7aadba9ed2f440"}
Sep 18 20:03:17.864833 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.864819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:17.866918 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.866900    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:03:17.873888 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:17.873868    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:03:18.045304 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.045271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.045537 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.045338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.045537 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.045398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.045537 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.045443    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.045537 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.045483    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgnx\" (UniqueName: \"kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.146456 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146429    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.146620 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.146620 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgnx\" (UniqueName: \"kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.146735 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.146735 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.147012 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.146993    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.147298 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.147279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.147360 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.147279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.148746 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.148727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.154412 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.154393    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgnx\" (UniqueName: \"kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-dllbl\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.175288 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.175264    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:18.447557 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.447530    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:03:18.450105 ip-10-0-143-198 kubenswrapper[2567]: W0918 20:03:18.450078    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21853fe9_4cf1_40b4_8b7a_6a80ac0d9c74.slice/crio-ae0def9672ab71d83146030728f54d01872488475d086c3f66420fa792e569fc WatchSource:0}: Error finding container ae0def9672ab71d83146030728f54d01872488475d086c3f66420fa792e569fc: Status 404 returned error can't find the container with id ae0def9672ab71d83146030728f54d01872488475d086c3f66420fa792e569fc
Sep 18 20:03:18.868256 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.868222    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerStarted","Data":"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"}
Sep 18 20:03:18.868643 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:18.868279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerStarted","Data":"ae0def9672ab71d83146030728f54d01872488475d086c3f66420fa792e569fc"}
Sep 18 20:03:20.502722 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:20.502684    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 18 20:03:20.507010 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:20.506987    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 18 20:03:21.883443 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:21.883403    2567 generic.go:356] "Generic (PLEG): container finished" podID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerID="a685fcde488a33378607ba0432ca4e24bcd2bf4612590308fff11d8a1820fc4c" exitCode=0
Sep 18 20:03:21.883763 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:21.883476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerDied","Data":"a685fcde488a33378607ba0432ca4e24bcd2bf4612590308fff11d8a1820fc4c"}
Sep 18 20:03:21.986226 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:21.986203    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:03:22.079608 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.079582    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csq8w\" (UniqueName: \"kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w\") pod \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") "
Sep 18 20:03:22.079785 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.079614    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") pod \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") "
Sep 18 20:03:22.079785 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.079654    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config\") pod \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") "
Sep 18 20:03:22.079785 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.079689    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert\") pod \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") "
Sep 18 20:03:22.079785 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.079721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location\") pod \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\" (UID: \"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a\") "
Sep 18 20:03:22.080070 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.080044    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:22.080115 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.080066    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:22.080115 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.080076    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:22.081730 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.081710    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w" (OuterVolumeSpecName: "kube-api-access-csq8w") pod "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a"). InnerVolumeSpecName "kube-api-access-csq8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:22.081838 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.081770    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" (UID: "45cdd521-d5fa-431d-9d4e-20c68e5cfd9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:22.180282 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.180259    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:03:22.180282 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.180279    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csq8w\" (UniqueName: \"kubernetes.io/projected/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-api-access-csq8w\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:03:22.180446 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.180288    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:03:22.180446 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.180297    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:03:22.180446 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.180306    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:03:22.889232 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.889203    2567 generic.go:356] "Generic (PLEG): container finished" podID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerID="bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0" exitCode=0
Sep 18 20:03:22.889647 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.889281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerDied","Data":"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"}
Sep 18 20:03:22.891413 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.891393    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4" event={"ID":"45cdd521-d5fa-431d-9d4e-20c68e5cfd9a","Type":"ContainerDied","Data":"d126a42210f6bce3ca5c8413ef621e489e2089f3072d1dc40e1c63c2bf964e2c"}
Sep 18 20:03:22.891518 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.891421    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"
Sep 18 20:03:22.891518 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.891441    2567 scope.go:117] "RemoveContainer" containerID="c73d56d28af6cb44901d9e0f83b37f9e6e83510660a3ec506a7aadba9ed2f440"
Sep 18 20:03:22.902511 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.902487    2567 scope.go:117] "RemoveContainer" containerID="a685fcde488a33378607ba0432ca4e24bcd2bf4612590308fff11d8a1820fc4c"
Sep 18 20:03:22.917298 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.917281    2567 scope.go:117] "RemoveContainer" containerID="66f75168a02591effb8ca63a0a862dc5383ce4da16927867e32dd9af7de550bd"
Sep 18 20:03:22.918909 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.918887    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:03:22.922546 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:22.922527    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-zgvp4"]
Sep 18 20:03:23.899190 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.899157    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerStarted","Data":"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"}
Sep 18 20:03:23.899667 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.899215    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerStarted","Data":"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"}
Sep 18 20:03:23.899667 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.899503    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:23.899776 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.899684    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:23.901001 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.900973    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:23.917988 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:23.917943    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podStartSLOduration=2.963769242 podStartE2EDuration="6.917928132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95415889s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:23.917006287 +0000 UTC m=+1501.696376869" watchObservedRunningTime="2026-09-18 20:03:23.917928132 +0000 UTC m=+1501.697298714"
Sep 18 20:03:24.340289 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:24.340260    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 20:03:24.343760 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:24.343741    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 20:03:24.353471 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:24.353448    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" path="/var/lib/kubelet/pods/45cdd521-d5fa-431d-9d4e-20c68e5cfd9a/volumes"
Sep 18 20:03:24.903121 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:24.903085    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:29.909043 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:29.909014    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:03:29.909535 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:29.909509    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:39.910191 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:39.910147    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:49.909499 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:49.909455    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:03:59.909539 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:03:59.909495    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:04:09.910193 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:09.910154    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:04:19.909597 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:19.909553    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:04:29.909567 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:29.909531    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:04:33.348132 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:33.348097    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 18 20:04:43.348527 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:43.348499    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:04:47.958913 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:47.958880    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:04:47.959321 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:47.959164    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container" containerID="cri-o://4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26" gracePeriod=30
Sep 18 20:04:47.959321 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:47.959212    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kube-rbac-proxy" containerID="cri-o://46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d" gracePeriod=30
Sep 18 20:04:48.254762 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:48.254684    2567 generic.go:356] "Generic (PLEG): container finished" podID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerID="46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d" exitCode=2
Sep 18 20:04:48.254879 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:48.254765    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerDied","Data":"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"}
Sep 18 20:04:49.903580 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:49.903539    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 18 20:04:52.098538 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.098517    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:04:52.195413 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195347    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert\") pod \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") "
Sep 18 20:04:52.195413 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195390    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location\") pod \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") "
Sep 18 20:04:52.195570 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195413    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config\") pod \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") "
Sep 18 20:04:52.195570 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195491    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kgnx\" (UniqueName: \"kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx\") pod \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") "
Sep 18 20:04:52.195651 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195576    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls\") pod \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\" (UID: \"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74\") "
Sep 18 20:04:52.195711 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" (UID: "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:52.195753 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195724    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" (UID: "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:52.195795 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195768    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" (UID: "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:52.195895 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195880    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-cabundle-cert\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.195940 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195902    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kserve-provision-location\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.195940 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.195915    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.197540 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.197507    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx" (OuterVolumeSpecName: "kube-api-access-2kgnx") pod "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" (UID: "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74"). InnerVolumeSpecName "kube-api-access-2kgnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:52.197637 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.197547    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" (UID: "21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:52.273216 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.273187    2567 generic.go:356] "Generic (PLEG): container finished" podID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerID="4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26" exitCode=0
Sep 18 20:04:52.273320 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.273244    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerDied","Data":"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"}
Sep 18 20:04:52.273320 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.273251    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"
Sep 18 20:04:52.273320 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.273305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl" event={"ID":"21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74","Type":"ContainerDied","Data":"ae0def9672ab71d83146030728f54d01872488475d086c3f66420fa792e569fc"}
Sep 18 20:04:52.273447 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.273342    2567 scope.go:117] "RemoveContainer" containerID="46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"
Sep 18 20:04:52.281597 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.281577    2567 scope.go:117] "RemoveContainer" containerID="4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"
Sep 18 20:04:52.289253 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.289237    2567 scope.go:117] "RemoveContainer" containerID="bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"
Sep 18 20:04:52.293459 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.293440    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:04:52.296620 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.296594    2567 scope.go:117] "RemoveContainer" containerID="46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"
Sep 18 20:04:52.296939 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:04:52.296914    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d\": container with ID starting with 46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d not found: ID does not exist" containerID="46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"
Sep 18 20:04:52.296939 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.296926    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kgnx\" (UniqueName: \"kubernetes.io/projected/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-kube-api-access-2kgnx\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.297054 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.296941    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d"} err="failed to get container status \"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d\": rpc error: code = NotFound desc = could not find container \"46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d\": container with ID starting with 46c3ff0ef1007562c0a14afea07d815ac092d33683f526376a8bbf89140d236d not found: ID does not exist"
Sep 18 20:04:52.297054 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.296962    2567 scope.go:117] "RemoveContainer" containerID="4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"
Sep 18 20:04:52.297054 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.296948    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74-proxy-tls\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.297536 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:04:52.297492    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26\": container with ID starting with 4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26 not found: ID does not exist" containerID="4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"
Sep 18 20:04:52.297536 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.297524    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26"} err="failed to get container status \"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26\": rpc error: code = NotFound desc = could not find container \"4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26\": container with ID starting with 4b270c04214d84869dd8098aa27cc856f0fc8f7a698c028a527d7cf3ec447e26 not found: ID does not exist"
Sep 18 20:04:52.297697 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.297549    2567 scope.go:117] "RemoveContainer" containerID="bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"
Sep 18 20:04:52.297850 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:04:52.297825    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0\": container with ID starting with bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0 not found: ID does not exist" containerID="bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"
Sep 18 20:04:52.297892 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.297861    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0"} err="failed to get container status \"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0\": rpc error: code = NotFound desc = could not find container \"bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0\": container with ID starting with bce1fd74e01d1b7f25e957add6bcbec9f96037d0b440c39f6a7bb353b14b85c0 not found: ID does not exist"
Sep 18 20:04:52.299284 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.299267    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-dllbl"]
Sep 18 20:04:52.352339 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:04:52.352317    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" path="/var/lib/kubelet/pods/21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74/volumes"
Sep 18 20:06:48.293908 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.293874    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jr2dr/must-gather-5lg4m"]
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294231    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kube-rbac-proxy"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294242    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kube-rbac-proxy"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294253    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294260    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294273    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="storage-initializer"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294278    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="storage-initializer"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294289    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294295    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294303    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kube-rbac-proxy"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294308    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kube-rbac-proxy"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294321    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="storage-initializer"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294326    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="storage-initializer"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294389    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294399    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kube-rbac-proxy"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294406    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cdd521-d5fa-431d-9d4e-20c68e5cfd9a" containerName="kserve-container"
Sep 18 20:06:48.294439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.294414    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21853fe9-4cf1-40b4-8b7a-6a80ac0d9c74" containerName="kube-rbac-proxy"
Sep 18 20:06:48.297462 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.297442    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.299449 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.299416    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr2dr\"/\"kube-root-ca.crt\""
Sep 18 20:06:48.299557 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.299449    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jr2dr\"/\"openshift-service-ca.crt\""
Sep 18 20:06:48.299557 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.299492    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jr2dr\"/\"default-dockercfg-gztqg\""
Sep 18 20:06:48.303857 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.303831    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr2dr/must-gather-5lg4m"]
Sep 18 20:06:48.361019 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.360988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzfr\" (UniqueName: \"kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.361169 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.361035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.462593 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.462548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzfr\" (UniqueName: \"kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.462800 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.462716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.463043 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.463021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.472116 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.472082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzfr\" (UniqueName: \"kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr\") pod \"must-gather-5lg4m\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") " pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.619781 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.619695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:06:48.886715 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.886692    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jr2dr/must-gather-5lg4m"]
Sep 18 20:06:48.888565 ip-10-0-143-198 kubenswrapper[2567]: W0918 20:06:48.888535    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f16b410_26f7_4701_b008_d8a8c50bdd94.slice/crio-84bbc2246e5b610c1d182aab651f8b76ea8f97016beb1644f35de46d2ec99fef WatchSource:0}: Error finding container 84bbc2246e5b610c1d182aab651f8b76ea8f97016beb1644f35de46d2ec99fef: Status 404 returned error can't find the container with id 84bbc2246e5b610c1d182aab651f8b76ea8f97016beb1644f35de46d2ec99fef
Sep 18 20:06:48.890384 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:48.890350    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:06:49.760194 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:49.760128    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" event={"ID":"9f16b410-26f7-4701-b008-d8a8c50bdd94","Type":"ContainerStarted","Data":"84bbc2246e5b610c1d182aab651f8b76ea8f97016beb1644f35de46d2ec99fef"}
Sep 18 20:06:55.798158 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:55.798113    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" event={"ID":"9f16b410-26f7-4701-b008-d8a8c50bdd94","Type":"ContainerStarted","Data":"a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21"}
Sep 18 20:06:55.798597 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:55.798180    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" event={"ID":"9f16b410-26f7-4701-b008-d8a8c50bdd94","Type":"ContainerStarted","Data":"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"}
Sep 18 20:06:55.812798 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:06:55.812748    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" podStartSLOduration=1.700978732 podStartE2EDuration="7.812730843s" totalImagesPullingTime="6.111752111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:55.811286785 +0000 UTC m=+1713.590657369" watchObservedRunningTime="2026-09-18 20:06:55.812730843 +0000 UTC m=+1713.592101428"
Sep 18 20:07:18.902649 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:18.902618    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerID="7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729" exitCode=0
Sep 18 20:07:18.903052 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:18.902690    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" event={"ID":"9f16b410-26f7-4701-b008-d8a8c50bdd94","Type":"ContainerDied","Data":"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"}
Sep 18 20:07:18.903052 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:18.902973    2567 scope.go:117] "RemoveContainer" containerID="7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"
Sep 18 20:07:19.485165 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:19.485138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jr2dr_must-gather-5lg4m_9f16b410-26f7-4701-b008-d8a8c50bdd94/gather/0.log"
Sep 18 20:07:22.722984 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:22.722949    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s98hl_aaab3569-3c45-4a03-bcd2-81b2b0a4f309/global-pull-secret-syncer/0.log"
Sep 18 20:07:22.865816 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:22.865786    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-csv52_b69094ee-26c9-4c0b-a6c5-042b65834156/konnectivity-agent/0.log"
Sep 18 20:07:22.974393 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:22.974288    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-198.ec2.internal_49ff6efb868a47e5add55da0b50fc28c/haproxy/0.log"
Sep 18 20:07:24.911261 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:24.911219    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jr2dr/must-gather-5lg4m"]
Sep 18 20:07:24.911822 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:24.911491    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jr2dr/must-gather-5lg4m" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="copy" containerID="cri-o://a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21" gracePeriod=2
Sep 18 20:07:24.913013 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:24.912978    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jr2dr/must-gather-5lg4m"]
Sep 18 20:07:24.913117 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:24.913043    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5lg4m\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jr2dr\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.142071 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.142049    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jr2dr_must-gather-5lg4m_9f16b410-26f7-4701-b008-d8a8c50bdd94/copy/0.log"
Sep 18 20:07:25.143539 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.143519    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.145638 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:25.145612    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5lg4m\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jr2dr\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.191229 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.191157    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output\") pod \"9f16b410-26f7-4701-b008-d8a8c50bdd94\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") "
Sep 18 20:07:25.191229 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.191214    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jzfr\" (UniqueName: \"kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr\") pod \"9f16b410-26f7-4701-b008-d8a8c50bdd94\" (UID: \"9f16b410-26f7-4701-b008-d8a8c50bdd94\") "
Sep 18 20:07:25.193315 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.193287    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9f16b410-26f7-4701-b008-d8a8c50bdd94" (UID: "9f16b410-26f7-4701-b008-d8a8c50bdd94"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:25.193410 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.193388    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr" (OuterVolumeSpecName: "kube-api-access-6jzfr") pod "9f16b410-26f7-4701-b008-d8a8c50bdd94" (UID: "9f16b410-26f7-4701-b008-d8a8c50bdd94"). InnerVolumeSpecName "kube-api-access-6jzfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:25.292344 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.292308    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzfr\" (UniqueName: \"kubernetes.io/projected/9f16b410-26f7-4701-b008-d8a8c50bdd94-kube-api-access-6jzfr\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:07:25.292344 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.292338    2567 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f16b410-26f7-4701-b008-d8a8c50bdd94-must-gather-output\") on node \"ip-10-0-143-198.ec2.internal\" DevicePath \"\""
Sep 18 20:07:25.936633 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.936606    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jr2dr_must-gather-5lg4m_9f16b410-26f7-4701-b008-d8a8c50bdd94/copy/0.log"
Sep 18 20:07:25.937075 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.936961    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerID="a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21" exitCode=143
Sep 18 20:07:25.937075 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.937017    2567 scope.go:117] "RemoveContainer" containerID="a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21"
Sep 18 20:07:25.937075 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.937020    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.939387 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:25.939347    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5lg4m\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jr2dr\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.945267 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.945250    2567 scope.go:117] "RemoveContainer" containerID="7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"
Sep 18 20:07:25.946700 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:25.946680    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5lg4m\" is forbidden: User \"system:node:ip-10-0-143-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jr2dr\": no relationship found between node 'ip-10-0-143-198.ec2.internal' and this object" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" pod="openshift-must-gather-jr2dr/must-gather-5lg4m"
Sep 18 20:07:25.956228 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.956212    2567 scope.go:117] "RemoveContainer" containerID="a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21"
Sep 18 20:07:25.956535 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:25.956517    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21\": container with ID starting with a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21 not found: ID does not exist" containerID="a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21"
Sep 18 20:07:25.956578 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.956546    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21"} err="failed to get container status \"a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21\": rpc error: code = NotFound desc = could not find container \"a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21\": container with ID starting with a77d3530ba3f2e6dfed297ecb4b44eeea1e21689707053752f4273942700da21 not found: ID does not exist"
Sep 18 20:07:25.956578 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.956562    2567 scope.go:117] "RemoveContainer" containerID="7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"
Sep 18 20:07:25.956812 ip-10-0-143-198 kubenswrapper[2567]: E0918 20:07:25.956793    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729\": container with ID starting with 7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729 not found: ID does not exist" containerID="7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"
Sep 18 20:07:25.956845 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:25.956818    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729"} err="failed to get container status \"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729\": rpc error: code = NotFound desc = could not find container \"7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729\": container with ID starting with 7d283500926e816b0c77601554629143d69db091be9159d87b4fef9ea40a6729 not found: ID does not exist"
Sep 18 20:07:26.353459 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.353425    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" path="/var/lib/kubelet/pods/9f16b410-26f7-4701-b008-d8a8c50bdd94/volumes"
Sep 18 20:07:26.848093 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.848063    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-nwq8p_a72572f6-da61-48e6-98cd-ab9d4478cd23/cluster-monitoring-operator/0.log"
Sep 18 20:07:26.877049 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.877021    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-gwxfz_f1ba49fe-4fb4-47ed-a640-7325e7f89518/kube-state-metrics/0.log"
Sep 18 20:07:26.897213 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.897189    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-gwxfz_f1ba49fe-4fb4-47ed-a640-7325e7f89518/kube-rbac-proxy-main/0.log"
Sep 18 20:07:26.924598 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.924573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-gwxfz_f1ba49fe-4fb4-47ed-a640-7325e7f89518/kube-rbac-proxy-self/0.log"
Sep 18 20:07:26.958388 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:26.958347    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b55d657d-pmjh5_f5bc57e6-387c-4b5b-9c42-57b4fbd9564d/metrics-server/0.log"
Sep 18 20:07:27.019507 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.019479    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mkpp_ae0aa986-cb47-49b5-a290-022d3637fb78/node-exporter/0.log"
Sep 18 20:07:27.042519 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.042497    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mkpp_ae0aa986-cb47-49b5-a290-022d3637fb78/kube-rbac-proxy/0.log"
Sep 18 20:07:27.066759 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.066732    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mkpp_ae0aa986-cb47-49b5-a290-022d3637fb78/init-textfile/0.log"
Sep 18 20:07:27.383284 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.383256    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/prometheus/0.log"
Sep 18 20:07:27.404318 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.404295    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/config-reloader/0.log"
Sep 18 20:07:27.431858 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.431830    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/thanos-sidecar/0.log"
Sep 18 20:07:27.456686 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.456656    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/kube-rbac-proxy-web/0.log"
Sep 18 20:07:27.484313 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.484289    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/kube-rbac-proxy/0.log"
Sep 18 20:07:27.507608 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.507585    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/kube-rbac-proxy-thanos/0.log"
Sep 18 20:07:27.527084 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.527064    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b1e08a4-21bd-49d1-aef7-94d2d7f5d2f6/init-config-reloader/0.log"
Sep 18 20:07:27.604113 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.604080    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9bffz_730a5f3d-ee26-4529-8c97-34970f6dc091/prometheus-operator-admission-webhook/0.log"
Sep 18 20:07:27.733048 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.732961    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/thanos-query/0.log"
Sep 18 20:07:27.753315 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.753291    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/kube-rbac-proxy-web/0.log"
Sep 18 20:07:27.778482 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.778452    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/kube-rbac-proxy/0.log"
Sep 18 20:07:27.803235 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.803208    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/prom-label-proxy/0.log"
Sep 18 20:07:27.830965 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.830944    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/kube-rbac-proxy-rules/0.log"
Sep 18 20:07:27.853953 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:27.853912    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84fb5f84b9-gnpxd_f6aeb7ed-56c2-4533-94d0-aa4b38d01961/kube-rbac-proxy-metrics/0.log"
Sep 18 20:07:29.627403 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:29.627313    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/2.log"
Sep 18 20:07:29.633967 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:29.633935    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-gkkdr_93f8637f-13ec-4134-ab92-45d02e00ce46/console-operator/3.log"
Sep 18 20:07:30.630156 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.630125    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-mcgs7_b6b0fcff-6aef-482c-a3ff-05e28e0cf311/volume-data-source-validator/0.log"
Sep 18 20:07:30.985718 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.985637    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"]
Sep 18 20:07:30.985984 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.985971    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="gather"
Sep 18 20:07:30.986026 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.985986    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="gather"
Sep 18 20:07:30.986026 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.985995    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="copy"
Sep 18 20:07:30.986026 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.986000    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="copy"
Sep 18 20:07:30.986123 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.986062    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="gather"
Sep 18 20:07:30.986123 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.986071    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f16b410-26f7-4701-b008-d8a8c50bdd94" containerName="copy"
Sep 18 20:07:30.989825 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.989803    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:30.991896 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.991876    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rlcn6\"/\"kube-root-ca.crt\""
Sep 18 20:07:30.992009 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.991908    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rlcn6\"/\"openshift-service-ca.crt\""
Sep 18 20:07:30.992406 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.992392    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rlcn6\"/\"default-dockercfg-dr5wp\""
Sep 18 20:07:30.998498 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:30.998476    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"]
Sep 18 20:07:31.039208 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.039182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-lib-modules\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.039385 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.039225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-proc\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.039385 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.039244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57bk\" (UniqueName: \"kubernetes.io/projected/cf77ee30-1492-4a68-a630-64d995154c3a-kube-api-access-v57bk\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.039385 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.039314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-podres\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.039385 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.039357    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-sys\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140587 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-lib-modules\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140757 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-proc\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140757 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57bk\" (UniqueName: \"kubernetes.io/projected/cf77ee30-1492-4a68-a630-64d995154c3a-kube-api-access-v57bk\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140757 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140690    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-proc\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140757 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-lib-modules\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140757 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-podres\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140992 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-podres\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140992 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-sys\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.140992 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.140861    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf77ee30-1492-4a68-a630-64d995154c3a-sys\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.147828 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.147805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57bk\" (UniqueName: \"kubernetes.io/projected/cf77ee30-1492-4a68-a630-64d995154c3a-kube-api-access-v57bk\") pod \"perf-node-gather-daemonset-z8f6r\" (UID: \"cf77ee30-1492-4a68-a630-64d995154c3a\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.300667 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.300634    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.533991 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.533962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7bsqr_06f430df-4b07-4547-b71e-68541a5134ab/dns/0.log"
Sep 18 20:07:31.554270 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.554187    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7bsqr_06f430df-4b07-4547-b71e-68541a5134ab/kube-rbac-proxy/0.log"
Sep 18 20:07:31.568214 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.568192    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"]
Sep 18 20:07:31.570775 ip-10-0-143-198 kubenswrapper[2567]: W0918 20:07:31.570748    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcf77ee30_1492_4a68_a630_64d995154c3a.slice/crio-11fd06559d7e696b941462e008c9783cd90854c61db4adb2f436add650b55612 WatchSource:0}: Error finding container 11fd06559d7e696b941462e008c9783cd90854c61db4adb2f436add650b55612: Status 404 returned error can't find the container with id 11fd06559d7e696b941462e008c9783cd90854c61db4adb2f436add650b55612
Sep 18 20:07:31.679780 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.679749    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-768dq_70ded459-00cc-4455-a8a4-a080de66c481/dns-node-resolver/0.log"
Sep 18 20:07:31.964100 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.964014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r" event={"ID":"cf77ee30-1492-4a68-a630-64d995154c3a","Type":"ContainerStarted","Data":"d5a4df39390a5358ed80c17e68341dc677964f815b0a9c5392283b6336dc25cd"}
Sep 18 20:07:31.964100 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.964067    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r" event={"ID":"cf77ee30-1492-4a68-a630-64d995154c3a","Type":"ContainerStarted","Data":"11fd06559d7e696b941462e008c9783cd90854c61db4adb2f436add650b55612"}
Sep 18 20:07:31.964296 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.964143    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:31.981025 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:31.980980    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r" podStartSLOduration=1.9809640929999999 podStartE2EDuration="1.980964093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:31.980198389 +0000 UTC m=+1749.759568971" watchObservedRunningTime="2026-09-18 20:07:31.980964093 +0000 UTC m=+1749.760334675"
Sep 18 20:07:32.215551 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:32.215466    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6qv7_a369c464-bec9-42e5-87ef-63208b3d90f6/node-ca/0.log"
Sep 18 20:07:33.392337 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.392303    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djj5j_11bdee4d-4727-49bd-91e4-8ba898621b9c/serve-healthcheck-canary/0.log"
Sep 18 20:07:33.761230 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.761148    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4vsvd_093c8031-69fd-4cde-84f0-cdd7614bdb8b/insights-operator/0.log"
Sep 18 20:07:33.761408 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.761252    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4vsvd_093c8031-69fd-4cde-84f0-cdd7614bdb8b/insights-operator/1.log"
Sep 18 20:07:33.849240 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.849209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lth96_10eec82f-5705-4e7f-845d-2ecbe2f09ff8/kube-rbac-proxy/0.log"
Sep 18 20:07:33.868187 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.868145    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lth96_10eec82f-5705-4e7f-845d-2ecbe2f09ff8/exporter/0.log"
Sep 18 20:07:33.889099 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:33.889075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lth96_10eec82f-5705-4e7f-845d-2ecbe2f09ff8/extractor/0.log"
Sep 18 20:07:36.465686 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:36.465648    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6vt92_01b9bebe-d3f5-4fe8-ab78-3b36d55eb802/s3-init/0.log"
Sep 18 20:07:36.481877 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:36.481849    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6vt92_01b9bebe-d3f5-4fe8-ab78-3b36d55eb802/download-hf-model/0.log"
Sep 18 20:07:37.978633 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:37.978605    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-z8f6r"
Sep 18 20:07:40.588400 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:40.588358    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-rmn48_af4c7473-7965-4e64-8c84-5e2ddbcf6a71/migrator/0.log"
Sep 18 20:07:40.608440 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:40.608417    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-rmn48_af4c7473-7965-4e64-8c84-5e2ddbcf6a71/graceful-termination/0.log"
Sep 18 20:07:40.968189 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:40.968158    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-csqxv_051ad5ae-9e24-48fa-940d-fe0ea43a9491/kube-storage-version-migrator-operator/1.log"
Sep 18 20:07:40.970048 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:40.970009    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-csqxv_051ad5ae-9e24-48fa-940d-fe0ea43a9491/kube-storage-version-migrator-operator/0.log"
Sep 18 20:07:41.842869 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.842840    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6vr48_06f1c675-1a73-4385-864a-65d59dde6ad4/kube-multus/0.log"
Sep 18 20:07:41.865296 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.865274    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:07:41.886439 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.886403    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/egress-router-binary-copy/0.log"
Sep 18 20:07:41.906408 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.906387    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/cni-plugins/0.log"
Sep 18 20:07:41.925742 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.925719    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/bond-cni-plugin/0.log"
Sep 18 20:07:41.945181 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.945162    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/routeoverride-cni/0.log"
Sep 18 20:07:41.966119 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.966097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/whereabouts-cni-bincopy/0.log"
Sep 18 20:07:41.991035 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:41.991011    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfw_1608a4d9-2d53-4b44-b6f1-b7159dce3a8f/whereabouts-cni/0.log"
Sep 18 20:07:42.429967 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:42.429943    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6p9z_695e01df-28bb-4aed-9e24-d53720c990fc/network-metrics-daemon/0.log"
Sep 18 20:07:42.453639 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:42.453539    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6p9z_695e01df-28bb-4aed-9e24-d53720c990fc/kube-rbac-proxy/0.log"
Sep 18 20:07:43.658039 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.658010    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/ovn-controller/0.log"
Sep 18 20:07:43.678642 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.678618    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/ovn-acl-logging/0.log"
Sep 18 20:07:43.711408 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.711383    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/kube-rbac-proxy-node/0.log"
Sep 18 20:07:43.736245 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.736221    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:07:43.754648 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.754632    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/northd/0.log"
Sep 18 20:07:43.775911 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.775894    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/nbdb/0.log"
Sep 18 20:07:43.801463 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.801441    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/sbdb/0.log"
Sep 18 20:07:43.858669 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:43.858646    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b7nd9_991a6b3a-db32-4391-ab67-00f063ea5999/ovnkube-controller/0.log"
Sep 18 20:07:45.060115 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:45.060089    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-hwh7h_aa6278a5-600b-45eb-a49e-efbadf6d76b4/check-endpoints/0.log"
Sep 18 20:07:45.081434 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:45.081410    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5qnvn_48cc6b9c-9b06-46ae-bbe7-2535006c2b4c/network-check-target-container/0.log"
Sep 18 20:07:45.962325 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:45.962296    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5gtzj_adcea9b5-b950-4d8b-a1e5-03c410160484/iptables-alerter/0.log"
Sep 18 20:07:46.615551 ip-10-0-143-198 kubenswrapper[2567]: I0918 20:07:46.615522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dcpds_9434d76d-f399-40a7-912b-bdd6e219800e/tuned/0.log"