Sep 18 19:11:56.205113 ip-10-0-140-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:11:56.205125 ip-10-0-140-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:11:56.205132 ip-10-0-140-94 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:11:56.205283 ip-10-0-140-94 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:12:06.267263 ip-10-0-140-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:12:06.267280 ip-10-0-140-94 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 45a2240bb5fa4e79820166aa6248a575 --
Sep 18 19:14:14.783997 ip-10-0-140-94 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:14:18.275075 ip-10-0-140-94 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:18.275075 ip-10-0-140-94 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:14:18.275075 ip-10-0-140-94 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:18.275075 ip-10-0-140-94 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:14:18.275075 ip-10-0-140-94 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:18.275841 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.275149    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:14:18.277992 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.277973    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.277992 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.277990    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.277995    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.277998    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278001    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278004    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278007    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278010    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278013    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278016    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278026    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278029    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278031    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278034    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278037    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278039    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278041    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278044    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278047    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278050    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278052    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.278054 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278055    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278057    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278060    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278063    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278066    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278069    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278072    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278075    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278078    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278080    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278083    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278085    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278088    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278092    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278096    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278099    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278101    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278104    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278107    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278109    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.278531 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278111    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278114    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278119    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278122    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278125    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278127    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278129    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278132    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278134    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278137    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278139    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278142    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278144    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278148    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278152    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278156    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278159    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278162    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278165    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.279028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278167    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278170    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278172    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278175    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278178    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278180    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278185    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278187    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278190    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278192    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278195    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278197    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278200    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278202    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278205    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278208    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278211    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278213    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278216    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278219    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.279493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278221    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278225    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278227    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278230    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278232    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278235    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278237    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278241    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278244    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278246    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278251    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278254    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278257    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278260    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278263    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278266    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278269    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278271    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278274    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278986    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.280009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278993    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278996    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.278999    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279001    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279004    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279007    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279010    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279012    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279015    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279018    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279021    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279024    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279026    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279029    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279032    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279034    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279037    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279040    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279043    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.280534 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279047    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279049    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279052    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279055    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279058    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279061    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279063    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279066    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279068    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279071    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279075    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279079    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279082    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279085    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279087    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279090    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279094    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279097    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279100    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.281028 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279102    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279105    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279107    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279110    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279113    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279115    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279118    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279120    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279123    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279125    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279128    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279130    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279133    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279136    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279138    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279141    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279144    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279147    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279149    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279151    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.281503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279154    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279157    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279161    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279164    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279167    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279170    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279173    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279176    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279178    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279181    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279184    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279187    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279189    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279192    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279195    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279197    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279199    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279202    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279205    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279208    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.282007 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279210    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279214    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279217    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279219    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279222    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279225    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279228    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279231    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279233    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279236    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279238    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279240    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279243    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279246    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279249    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279251    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279254    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279257    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279259    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279261    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.282530 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279670    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279677    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279680    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279683    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279685    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279688    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279691    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279693    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279696    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279700    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279702    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279705    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279707    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279710    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279713    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279715    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279718    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279722    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279725    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279728    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.283024 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279730    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279733    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279735    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279738    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279740    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279743    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279745    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279747    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279750    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279753    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279755    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279757    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279760    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279765    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279767    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279770    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279772    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279775    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279777    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279780    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.283503 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279783    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279788    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279791    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279793    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279796    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279798    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279801    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279805    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279808    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279813    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279815    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279818    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279821    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279823    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279826    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279828    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279831    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279834    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279836    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.284006 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279838    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279841    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279843    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279846    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279848    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279850    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279854    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279856    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279859    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279861    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279864    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279866    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279868    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279871    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279873    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279877    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279881    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279884    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279886    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.284493 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279889    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279892    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279894    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279898    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279900    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279903    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279905    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279908    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279910    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279913    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279915    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279918    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279920    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279923    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279925    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279928    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279930    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279933    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279935    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.284970 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279939    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.279942    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280015    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280051    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280057    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280061    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280070    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280073    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280077    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280081    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280084    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280087    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280091    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280094    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280097    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280100    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280104    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280107    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280110    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280112    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280115    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280118    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280121    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280124    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:14:18.285418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280127    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280131    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280134    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280137    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280140    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280144    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280147    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280149    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280153    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280156    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280160    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280163    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280166    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280168    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280171    2584 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280174    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280178    2584 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280181    2584 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280183    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280186    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280190    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280193    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280196    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280199    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280203    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:14:18.286005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280206    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280209    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280211    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280214    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280217    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280220    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280222    2584 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280226    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280229    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280232    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280235    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280238    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280241    2584 flags.go:64] FLAG: --help="false"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280244    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280247    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280250    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280253    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280256    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280259    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280262    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280265    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280267    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280270    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:14:18.286611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280273    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280276    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280278    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280281    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280283    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280287    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280289    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280292    2584 flags.go:64] FLAG: --lock-file=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280296    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280299    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280301    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280307    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280309    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280312    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280315    2584 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280317    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280321    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280323    2584 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280326    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280330    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280333    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280337    2584 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280339    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280342    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280345    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:14:18.287205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280348    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280351    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280354    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280357    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280363    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280366    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280369    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280372    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280375    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280379    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280382    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280385    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280388    2584 flags.go:64] FLAG: --port="10250"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280392    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280395    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c689e80d2b837ad3"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280398    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280404    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280407    2584 flags.go:64] FLAG: --register-node="true"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280409    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280413    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280425    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280430    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280433    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280436    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280440    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:14:18.287824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280443    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280446    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280448    2584 flags.go:64] FLAG: --runonce="false"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280451    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280454    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280457    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280460    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280463    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280466    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280469    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280472    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280475    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280478    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280481    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280484    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280487    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280489    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280492    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280497    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280500    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280503    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280511    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280513    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280517    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280520    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:14:18.288625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280523    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280526    2584 flags.go:64] FLAG: --v="2"
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280530    2584 flags.go:64] FLAG: --version="false"
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280534    2584 flags.go:64] FLAG: --vmodule=""
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280538    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.280541    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280701    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280705    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280709    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280712    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280715    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280717    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280720    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280724    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280727    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280730    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280732    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280735    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280737    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280740    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280743    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.289453 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280745    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280748    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280750    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280753    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280755    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280758    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280760    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280763    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280766    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280769    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280772    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280774    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280777    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280779    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280782    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280785    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280787    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280790    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280792    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280795    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.290193 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280798    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280800    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280803    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280805    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280809    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280811    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280814    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280816    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280819    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280821    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280824    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280826    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280829    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280831    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280834    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280836    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280839    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280842    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280844    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280847    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.290689 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280849    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280852    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280855    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280858    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280861    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280863    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280866    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280869    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280871    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280874    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280876    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280879    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280881    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280884    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280887    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280891    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280896    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280898    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280901    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.291223 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280904    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280906    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280910    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280913    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280916    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280919    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280922    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280925    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280928    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280930    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280933    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280936    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280939    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280941    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280944    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280947    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280949    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280952    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280954    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.291707 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280957    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280960    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280963    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280966    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280969    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.280971    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.281454    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.290716    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.290731    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.290750    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.290759    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290814    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290819    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290823    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.292172 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290827    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290830    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290832    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290835    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290838    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290841    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290843    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290846    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290849    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290851    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290854    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290856    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290859    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290861    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290864    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290866    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290869    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290872    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290874    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290877    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.292570 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290880    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290882    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290885    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290887    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290890    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290892    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290896    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290899    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290901    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290904    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290908    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290912    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290915    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290917    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290920    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290923    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290925    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290928    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290930    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.293057 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290933    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290936    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290938    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290941    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290944    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290947    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290949    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290952    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290955    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290957    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290960    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290963    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290965    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290968    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290970    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290973    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290975    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290977    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290980    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290983    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.293516 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290986    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290989    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290992    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290995    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.290997    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291000    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291003    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291005    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291009    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291012    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291015    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291018    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291021    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291024    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291026    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291029    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291032    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291035    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291038    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.294071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291041    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291043    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291046    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291049    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291051    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291054    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291056    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291059    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291061    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291064    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291067    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291069    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291073    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291077    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291081    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291084    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291087    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.294535 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291090    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.291095    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291205    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291210    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291213    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291216    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291218    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291222    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291227    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291230    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291233    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291235    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291238    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291241    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291243    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:18.294986 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291246    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291248    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291251    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291253    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291256    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291258    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291261    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291263    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291266    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291269    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291272    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291275    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291278    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291281    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291284    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291287    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291289    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291292    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291294    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:18.295351 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291296    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291299    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291301    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291303    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291306    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291308    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291311    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291315    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291318    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291320    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291323    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291326    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291328    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291332    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291335    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291337    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291340    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291342    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291344    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:18.295825 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291347    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291349    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291352    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291354    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291356    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291359    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291362    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291365    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291367    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291370    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291373    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291375    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291378    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291380    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291382    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291385    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291387    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291389    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291392    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291395    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291397    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:18.296271 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291399    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291402    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291404    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291406    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291409    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291411    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291415    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291417    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291420    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291422    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291424    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291427    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291429    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291432    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291434    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291436    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291439    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291442    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291444    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:18.296794 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291447    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291449    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291452    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291454    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291457    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291459    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291461    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:18.291464    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.291469    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.291577    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.293625    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.294671    2584 server.go:1019] "Starting client certificate rotation"
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.294772    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:18.297276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.295524    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:18.297595 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.296927    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:18.298809 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.298787    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:18.313090 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.313071    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:14:18.314714 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.314698    2584 log.go:25] "Validated CRI v1 image API"
Sep 18 19:14:18.315548 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.315527    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:14:18.319837 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.319819    2584 fs.go:135] Filesystem UUIDs: map[566118e9-e6fe-4dca-8f0b-83f5730b2cf1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a6a4951e-e322-4f39-af2a-6d5e79bbd7ee:/dev/nvme0n1p4]
Sep 18 19:14:18.319898 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.319836    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:14:18.325232 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.325121    2584 manager.go:216] Machine: {Timestamp:2026-09-18 19:14:18.324654688 +0000 UTC m=+2.043311980 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102047 MemoryCapacity:32812142592 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242928a330f6b45816935e6071909d SystemUUID:ec242928-a330-f6b4-5816-935e6071909d BootID:45a2240b-b5fa-4e79-8201-66aa6248a575 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406069248 Type:vfs Inodes:4005388 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:e2:88:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:e2:88:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:e4:e8:f0:3c:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812142592 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:14:18.325232 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.325218    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:14:18.325388 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.325307    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:14:18.326823 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.326791    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:14:18.326943 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.326823    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:14:18.327932 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.327922    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:14:18.328008 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.327935    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:14:18.328008 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.327942    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:18.328008 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.327952    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:18.328083 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.328013    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:14:18.331329 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331317    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:18.331428 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331419    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:14:18.331464 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331450    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:14:18.331464 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331459    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:14:18.331534 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331473    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:14:18.331534 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331481    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:14:18.331534 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331489    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:14:18.331798 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331787    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:18.331839 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.331812    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:18.334116 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.334101    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:14:18.335766 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.335753    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:14:18.335825 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.335783    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:14:18.336849 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336829    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:14:18.336849 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336852    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336860    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336865    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336877    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336885    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336899    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336904    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336913    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336920    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336928    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:14:18.336963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336935    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:14:18.337222 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336970    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:14:18.337222 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.336978    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:14:18.340451 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.340431    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:18.340759 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.340745    2584 server.go:1268] "Started kubelet"
Sep 18 19:14:18.340850 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.340826    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:14:18.340896 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.340833    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:14:18.340930 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.340896    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:14:18.341656 ip-10-0-140-94 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:14:18.342672 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.342357    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:14:18.342672 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.342442    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:14:18.346029 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.345998    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:14:18.348383 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.348368    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:14:18.348460 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.348385    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:18.348697 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.348555    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.348942 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.348926    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:14:18.349038 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349020    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:14:18.349138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.348933    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:14:18.349138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349124    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:14:18.349138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349131    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:14:18.349275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349139    2584 factory.go:55] Registering systemd factory
Sep 18 19:14:18.349275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349155    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:14:18.349388 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349372    2584 factory.go:153] Registering CRI-O factory
Sep 18 19:14:18.349388 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349388    2584 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:14:18.349525 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349461    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:14:18.349525 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349480    2584 factory.go:103] Registering Raw factory
Sep 18 19:14:18.349525 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349494    2584 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:14:18.349854 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.349840    2584 manager.go:349] Starting recovery of all containers
Sep 18 19:14:18.375215 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.375184    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9ncz"
Sep 18 19:14:18.375455 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.375426    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:14:18.375652 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.375614    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:14:18.376028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.376007    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:14:18.379368 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.379309    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9ncz"
Sep 18 19:14:18.380091 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.380065    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:14:18.380294 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.380116    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:14:18.381133 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.380090    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-94.ec2.internal.18d68032f901db51  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-94.ec2.internal,UID:ip-10-0-140-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-94.ec2.internal,},FirstTimestamp:2026-09-18 19:14:18.340719441 +0000 UTC m=+2.059376734,LastTimestamp:2026-09-18 19:14:18.340719441 +0000 UTC m=+2.059376734,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-94.ec2.internal,}"
Sep 18 19:14:18.384710 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.384693    2584 manager.go:354] Recovery completed
Sep 18 19:14:18.388366 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.388353    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.390111 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390096    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.390189 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390121    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.390189 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390133    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.390690 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390678    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:14:18.390690 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390687    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.390690 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390691    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:14:18.390788 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.390705    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:18.391285 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.391269    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.391363 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.391311    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.391363 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.391327    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.392602 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.392589    2584 policy_none.go:49] "None policy: Start"
Sep 18 19:14:18.392715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.392604    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:14:18.392715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.392616    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:14:18.394680 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.394670    2584 policy_none.go:47] "Start"
Sep 18 19:14:18.430327 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430208    2584 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.430360    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430373    2584 server.go:87] "Starting device plugin registration server"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430627    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430668    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430749    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430870    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.430877    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.431324    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:14:18.448106 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.431373    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.503405 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.503378    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:14:18.504748 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.504729    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:14:18.504832 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.504776    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:14:18.504832 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.504798    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:14:18.504918 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.504834    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:14:18.507802 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.507784    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:18.531037 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.530987    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.533822 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.533797    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.533916 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.533830    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.533916 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.533844    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.533916 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.533881    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.542387 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.542373    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.542451 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.542392    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-94.ec2.internal\": node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.576167 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.576145    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.605846 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.605821    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"]
Sep 18 19:14:18.605920 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.605878    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.606607 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.606587    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.606726 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.606618    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.606726 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.606631    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.608872 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.608857    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.608993 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.608968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.609041 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609014    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.609685 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609670    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.609769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609685    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.609769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609698    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.609769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609706    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.609769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609713    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.609769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.609715    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.611877 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.611864    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.611938 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.611887    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:18.612666 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.612619    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:18.612666 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.612666    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:18.612782 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.612679    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:18.636198 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.636181    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-94.ec2.internal\" not found" node="ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.640884 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.640869    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-94.ec2.internal\" not found" node="ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.651760 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.651744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.651841 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.651767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.651841 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.651784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc46e81cf898f71be4cf1f3edbfc59bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"bc46e81cf898f71be4cf1f3edbfc59bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.676804 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.676784    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.752118 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.752182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.752182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc46e81cf898f71be4cf1f3edbfc59bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"bc46e81cf898f71be4cf1f3edbfc59bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.752182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc46e81cf898f71be4cf1f3edbfc59bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-94.ec2.internal\" (UID: \"bc46e81cf898f71be4cf1f3edbfc59bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.752284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.752284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.752196    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f03d5f60c7936b409a6408da67cd43c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal\" (UID: \"0f03d5f60c7936b409a6408da67cd43c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.777607 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.777588    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.877737 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.877690    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:18.940377 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.940341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.944266 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:18.944246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:18.978295 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:18.978269    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:19.078755 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:19.078722    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:19.179321 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:19.179265    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:19.270154 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.270132    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:19.279498 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:19.279480    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:19.294662 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.294630    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:19.294783 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.294767    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:19.294783 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.294769    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:19.380063 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:19.380038    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-94.ec2.internal\" not found"
Sep 18 19:14:19.384848 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.384815    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:18 +0000 UTC" deadline="2028-03-15 19:30:24.378406988 +0000 UTC"
Sep 18 19:14:19.384848 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.384845    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13056h16m4.993564504s"
Sep 18 19:14:19.391702 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.391683    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:19.448888 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.448837    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:19.466261 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.466242    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:19.467187 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.467171    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal"
Sep 18 19:14:19.478442 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.478421    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:19.564406 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:19.564382    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc46e81cf898f71be4cf1f3edbfc59bf.slice/crio-028dfe38b963a0337bfa769670129b0f49de745e388076bd72111211ef8f1e70 WatchSource:0}: Error finding container 028dfe38b963a0337bfa769670129b0f49de745e388076bd72111211ef8f1e70: Status 404 returned error can't find the container with id 028dfe38b963a0337bfa769670129b0f49de745e388076bd72111211ef8f1e70
Sep 18 19:14:19.564885 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:19.564861    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f03d5f60c7936b409a6408da67cd43c.slice/crio-a2b01f76a51382f8a7ebcbce8a33fdf13a1c276e54b03c9236b243ab00d385f0 WatchSource:0}: Error finding container a2b01f76a51382f8a7ebcbce8a33fdf13a1c276e54b03c9236b243ab00d385f0: Status 404 returned error can't find the container with id a2b01f76a51382f8a7ebcbce8a33fdf13a1c276e54b03c9236b243ab00d385f0
Sep 18 19:14:19.569608 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.569593    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:19.747120 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:19.747077    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:20.333060 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.333028    2584 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:20.350910 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.350170    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:20.356475 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.356453    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:20.357070 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.357030    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fs5tv","kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal","openshift-multus/network-metrics-daemon-l9829","openshift-network-diagnostics/network-check-target-c6td6","openshift-network-operator/iptables-alerter-lklth","openshift-ovn-kubernetes/ovnkube-node-j5x2c","kube-system/global-pull-secret-syncer-7thst","openshift-cluster-node-tuning-operator/tuned-r9rz9","openshift-dns/node-resolver-8jxxz","openshift-image-registry/node-ca-62z5k","openshift-multus/multus-additional-cni-plugins-fsx2n","openshift-multus/multus-qzkxt"]
Sep 18 19:14:20.359403 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.359380    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.361548 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.361527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.362273 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.362254    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9f6lc\""
Sep 18 19:14:20.362853 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.362833    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.363717 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.363696    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.366041 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.366021    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.368721 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.368453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.368721 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.368531    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:20.370737 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.370716    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:20.370839 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.370783    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:20.372916 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.372900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.372995 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.372954    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:20.375142 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.375119    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.377663 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.377625    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.380657 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.380499    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.383489 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.383468    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.385805 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.385656    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:20.386054 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.386039    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.386570 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.386546    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.386835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.386819    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.386936 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.386861    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:20.387108 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.387060    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:20.387192 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.387112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:20.400412 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.400393    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.400511 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.400401    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.401866 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.401846    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:20.402055 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.402041    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:20.405876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.405853    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.405963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.405860    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ch9ws\""
Sep 18 19:14:20.406023 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.405963    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kr9kd\""
Sep 18 19:14:20.420152 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.419996    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.420464 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420332    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.420953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420688    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.420953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420754    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:20.420953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420772    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrlpz\""
Sep 18 19:14:20.420953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.420953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.420890    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:20.421155 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.421068    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:20.427237 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427217    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5qr9\""
Sep 18 19:14:20.427237 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427230    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.427376 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427334    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:20.427461 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427445    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.427587 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:20.427587 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427214    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:20.427764 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427748    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r2rhm"
Sep 18 19:14:20.427825 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427766    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r2rhm"
Sep 18 19:14:20.427943 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.427927    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:20.428043 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.428023    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:20.435146 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.435124    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:20.436841 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.436494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kgzbw\""
Sep 18 19:14:20.438311 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.438294    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzmsw\""
Sep 18 19:14:20.438558 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.438540    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tnnmq\""
Sep 18 19:14:20.438800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.438765    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fr685\""
Sep 18 19:14:20.438892 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.438821    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:20.450225 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.450199    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:20.461778 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461755    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.461883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461788    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2b55544-874b-4f30-baa8-9288cde4cf37-host\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.461883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461805    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.461883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovn-node-metrics-cert\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.461883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.461883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461875    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52f7b\" (UniqueName: \"kubernetes.io/projected/d2b55544-874b-4f30-baa8-9288cde4cf37-kube-api-access-52f7b\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461916    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-var-lib-kubelet\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.461989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-systemd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-netd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cnibin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52cq\" (UniqueName: \"kubernetes.io/projected/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-kube-api-access-l52cq\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxt9\" (UniqueName: \"kubernetes.io/projected/be4be787-37a0-4434-bfc7-06824039ef7e-kube-api-access-cxxt9\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.462123 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462111    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-cnibin\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-bin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-run\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-tmp\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462285    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmflw\" (UniqueName: \"kubernetes.io/projected/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-kube-api-access-gmflw\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462359    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-socket-dir-parent\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-netns\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-config\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48a0cfb3-668d-4396-b407-133871aa0d41-hosts-file\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysconfig\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.462532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c26541-181f-4238-9063-4f2fc8fbb422-iptables-alerter-script\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462553    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-system-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-kubelet-config\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-conf\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-env-overrides\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462675    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-k8s-cni-cncf-io\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-multus\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-conf-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462741    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-dbus\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-kubernetes\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462787    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/f6c26541-181f-4238-9063-4f2fc8fbb422-kube-api-access-d92rr\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-slash\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-kubelet\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-etc-kubernetes\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-device-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.463284 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-kubelet\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-netns\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.462988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-log-socket\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-script-lib\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-daemon-config\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c26541-181f-4238-9063-4f2fc8fbb422-host-slash\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463121    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-ovn\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463143    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-hostroot\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwqf\" (UniqueName: \"kubernetes.io/projected/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-kube-api-access-7qwqf\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-node-log\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463210    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef57494-7501-4137-8770-70f06d6aa386-agent-certs\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57494-7501-4137-8770-70f06d6aa386-konnectivity-ca\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-system-cni-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.463835 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z52l\" (UniqueName: \"kubernetes.io/projected/0f46f69a-eedc-4d12-9a53-96baae2780b6-kube-api-access-9z52l\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a0cfb3-668d-4396-b407-133871aa0d41-tmp-dir\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463393    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfjd\" (UniqueName: \"kubernetes.io/projected/48a0cfb3-668d-4396-b407-133871aa0d41-kube-api-access-tcfjd\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463418    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-modprobe-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmhw\" (UniqueName: \"kubernetes.io/projected/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kube-api-access-npmhw\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-etc-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cni-binary-copy\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463509    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-lib-modules\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-var-lib-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-sys\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-host\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-systemd-units\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.464499 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-bin\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.465327 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-systemd\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.465327 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-etc-tuned\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.465327 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463795    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2b55544-874b-4f30-baa8-9288cde4cf37-serviceca\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.465327 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.463820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-multus-certs\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.510439 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.510384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"0f03d5f60c7936b409a6408da67cd43c","Type":"ContainerStarted","Data":"a2b01f76a51382f8a7ebcbce8a33fdf13a1c276e54b03c9236b243ab00d385f0"}
Sep 18 19:14:20.511860 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.511837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" event={"ID":"bc46e81cf898f71be4cf1f3edbfc59bf","Type":"ContainerStarted","Data":"028dfe38b963a0337bfa769670129b0f49de745e388076bd72111211ef8f1e70"}
Sep 18 19:14:20.564755 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-systemd\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.564864 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564775    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-etc-tuned\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.564864 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564795    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2b55544-874b-4f30-baa8-9288cde4cf37-serviceca\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.564864 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-systemd\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.564864 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-multus-certs\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.564864 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564862    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564871    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-multus-certs\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2b55544-874b-4f30-baa8-9288cde4cf37-host\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovn-node-metrics-cert\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.564954    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52f7b\" (UniqueName: \"kubernetes.io/projected/d2b55544-874b-4f30-baa8-9288cde4cf37-kube-api-access-52f7b\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.564982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-var-lib-kubelet\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.565029    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:21.064998386 +0000 UTC m=+4.783655688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.565067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-systemd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-netd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565082    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565136    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cnibin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-netd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.565326    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.565378    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:21.065361724 +0000 UTC m=+4.784019022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-systemd\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-var-lib-kubelet\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cnibin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2b55544-874b-4f30-baa8-9288cde4cf37-serviceca\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.565508 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52cq\" (UniqueName: \"kubernetes.io/projected/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-kube-api-access-l52cq\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxt9\" (UniqueName: \"kubernetes.io/projected/be4be787-37a0-4434-bfc7-06824039ef7e-kube-api-access-cxxt9\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-cnibin\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-bin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-run\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-cnibin\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-tmp\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565690    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565750    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmflw\" (UniqueName: \"kubernetes.io/projected/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-kube-api-access-gmflw\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565770    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-socket-dir-parent\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-netns\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2b55544-874b-4f30-baa8-9288cde4cf37-host\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565467    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.566148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565993    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-bin\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566081    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-socket-dir-parent\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-run\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.565827    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-config\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-netns\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48a0cfb3-668d-4396-b407-133871aa0d41-hosts-file\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysconfig\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566218    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c26541-181f-4238-9063-4f2fc8fbb422-iptables-alerter-script\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-system-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48a0cfb3-668d-4396-b407-133871aa0d41-hosts-file\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566273    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-kubelet-config\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566299    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-conf\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysconfig\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-env-overrides\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-k8s-cni-cncf-io\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-kubelet-config\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-multus\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.566961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-conf-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566409    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-system-cni-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-dbus\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-kubernetes\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566475    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/f6c26541-181f-4238-9063-4f2fc8fbb422-kube-api-access-d92rr\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566509    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-slash\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-kubelet\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-etc-kubernetes\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-device-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-run-k8s-cni-cncf-io\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566666    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-kubelet\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566684    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-config\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566690    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-cni-multus\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-netns\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.567791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6e38969-99fd-4925-9313-6ef4ac0674c2-dbus\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566733    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-netns\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-env-overrides\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-kubelet\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-kubernetes\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566815    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-etc-kubernetes\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-log-socket\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566846    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-conf\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-sysctl-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-script-lib\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-daemon-config\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-conf-dir\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-device-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-log-socket\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.568528 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c26541-181f-4238-9063-4f2fc8fbb422-host-slash\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.566986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-host-var-lib-kubelet\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567018    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c26541-181f-4238-9063-4f2fc8fbb422-iptables-alerter-script\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-slash\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-ovn\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c26541-181f-4238-9063-4f2fc8fbb422-host-slash\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-run-ovn\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567201    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-hostroot\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwqf\" (UniqueName: \"kubernetes.io/projected/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-kube-api-access-7qwqf\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-node-log\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567256    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-hostroot\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef57494-7501-4137-8770-70f06d6aa386-agent-certs\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57494-7501-4137-8770-70f06d6aa386-konnectivity-ca\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-node-log\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-system-cni-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.569238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z52l\" (UniqueName: \"kubernetes.io/projected/0f46f69a-eedc-4d12-9a53-96baae2780b6-kube-api-access-9z52l\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a0cfb3-668d-4396-b407-133871aa0d41-tmp-dir\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfjd\" (UniqueName: \"kubernetes.io/projected/48a0cfb3-668d-4396-b407-133871aa0d41-kube-api-access-tcfjd\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-modprobe-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmhw\" (UniqueName: \"kubernetes.io/projected/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kube-api-access-npmhw\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-etc-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-multus-daemon-config\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovnkube-script-lib\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567564    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cni-binary-copy\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-etc-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567752    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f46f69a-eedc-4d12-9a53-96baae2780b6-system-cni-dir\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567960    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-etc-modprobe-d\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.567996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-lib-modules\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef57494-7501-4137-8770-70f06d6aa386-konnectivity-ca\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.569947 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-var-lib-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-sys\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-host\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568140    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-cni-binary-copy\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-systemd-units\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a0cfb3-668d-4396-b407-133871aa0d41-tmp-dir\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568254    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-host\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-sys\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568270    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-bin\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-systemd-units\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-var-lib-openvswitch\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.570552 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568367    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-host-cni-bin\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.568364    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be4be787-37a0-4434-bfc7-06824039ef7e-lib-modules\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.574088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-tmp\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.574119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/be4be787-37a0-4434-bfc7-06824039ef7e-etc-tuned\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.574353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-ovn-node-metrics-cert\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.574422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef57494-7501-4137-8770-70f06d6aa386-agent-certs\") pod \"konnectivity-agent-fs5tv\" (UID: \"7ef57494-7501-4137-8770-70f06d6aa386\") " pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.576800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.574487    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f46f69a-eedc-4d12-9a53-96baae2780b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.586215 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.586165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52f7b\" (UniqueName: \"kubernetes.io/projected/d2b55544-874b-4f30-baa8-9288cde4cf37-kube-api-access-52f7b\") pod \"node-ca-62z5k\" (UID: \"d2b55544-874b-4f30-baa8-9288cde4cf37\") " pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:20.592965 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.592944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmflw\" (UniqueName: \"kubernetes.io/projected/1d130a7b-ffca-47cf-9c74-7e4632c70b4a-kube-api-access-gmflw\") pod \"ovnkube-node-j5x2c\" (UID: \"1d130a7b-ffca-47cf-9c74-7e4632c70b4a\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.593450 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.593413    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwqf\" (UniqueName: \"kubernetes.io/projected/393f2e3b-3dbb-4d68-b145-ba860b2a0c08-kube-api-access-7qwqf\") pod \"multus-qzkxt\" (UID: \"393f2e3b-3dbb-4d68-b145-ba860b2a0c08\") " pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.598423 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.598404    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:20.598423 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.598425    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:20.598549 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.598434    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:20.598549 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:20.598484    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:21.098468271 +0000 UTC m=+4.817125570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:20.598866 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.598844    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52cq\" (UniqueName: \"kubernetes.io/projected/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-kube-api-access-l52cq\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:20.608080 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.608047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/f6c26541-181f-4238-9063-4f2fc8fbb422-kube-api-access-d92rr\") pod \"iptables-alerter-lklth\" (UID: \"f6c26541-181f-4238-9063-4f2fc8fbb422\") " pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.610514 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.610495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmhw\" (UniqueName: \"kubernetes.io/projected/36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf-kube-api-access-npmhw\") pod \"aws-ebs-csi-driver-node-xzvdl\" (UID: \"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.611978 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.611957    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxt9\" (UniqueName: \"kubernetes.io/projected/be4be787-37a0-4434-bfc7-06824039ef7e-kube-api-access-cxxt9\") pod \"tuned-r9rz9\" (UID: \"be4be787-37a0-4434-bfc7-06824039ef7e\") " pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.613051 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.613031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z52l\" (UniqueName: \"kubernetes.io/projected/0f46f69a-eedc-4d12-9a53-96baae2780b6-kube-api-access-9z52l\") pod \"multus-additional-cni-plugins-fsx2n\" (UID: \"0f46f69a-eedc-4d12-9a53-96baae2780b6\") " pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.613138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.613097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfjd\" (UniqueName: \"kubernetes.io/projected/48a0cfb3-668d-4396-b407-133871aa0d41-kube-api-access-tcfjd\") pod \"node-resolver-8jxxz\" (UID: \"48a0cfb3-668d-4396-b407-133871aa0d41\") " pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.619170 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.619154    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:20.671698 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.671678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9rz9"
Sep 18 19:14:20.685177 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.685160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fsx2n"
Sep 18 19:14:20.697362 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.697338    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl"
Sep 18 19:14:20.701833 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.701816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qzkxt"
Sep 18 19:14:20.708814 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.708787    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lklth"
Sep 18 19:14:20.716695 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.716675    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:20.723372 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.723345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:20.730859 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.730838    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jxxz"
Sep 18 19:14:20.737051 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:20.737037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-62z5k"
Sep 18 19:14:21.072991 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.072963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:21.072991 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.072996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:21.073221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.073107    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:21.073221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.073114    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:21.073221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.073159    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:22.07314343 +0000 UTC m=+5.791800711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:21.073221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.073174    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:22.073168182 +0000 UTC m=+5.791825460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:21.091769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.091749    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:21.173319 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.173277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:21.173463 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.173414    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:21.173463 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.173436    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:21.173463 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.173449    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:21.173617 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.173510    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:22.173492585 +0000 UTC m=+5.892149870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:21.413059 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.413032    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c26541_181f_4238_9063_4f2fc8fbb422.slice/crio-35a9a59363ab65bc72ebfc428ccd7485837ac212b4dbbf9157486e9ef1769a06 WatchSource:0}: Error finding container 35a9a59363ab65bc72ebfc428ccd7485837ac212b4dbbf9157486e9ef1769a06: Status 404 returned error can't find the container with id 35a9a59363ab65bc72ebfc428ccd7485837ac212b4dbbf9157486e9ef1769a06
Sep 18 19:14:21.414708 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.414684    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f46f69a_eedc_4d12_9a53_96baae2780b6.slice/crio-3f7c234ed4ad18b1b2d909a933b71885193cf90bde6ec42c48ea70cc46c2137b WatchSource:0}: Error finding container 3f7c234ed4ad18b1b2d909a933b71885193cf90bde6ec42c48ea70cc46c2137b: Status 404 returned error can't find the container with id 3f7c234ed4ad18b1b2d909a933b71885193cf90bde6ec42c48ea70cc46c2137b
Sep 18 19:14:21.431057 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.431027    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:20 +0000 UTC" deadline="2028-02-17 13:58:18.431898791 +0000 UTC"
Sep 18 19:14:21.431138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.431052    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12402h43m57.000850401s"
Sep 18 19:14:21.436771 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.436752    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4be787_37a0_4434_bfc7_06824039ef7e.slice/crio-b3bfa62cfa26dfb8bc0e5492080217b5a23c21c728ea3296bda8aede11ceecd6 WatchSource:0}: Error finding container b3bfa62cfa26dfb8bc0e5492080217b5a23c21c728ea3296bda8aede11ceecd6: Status 404 returned error can't find the container with id b3bfa62cfa26dfb8bc0e5492080217b5a23c21c728ea3296bda8aede11ceecd6
Sep 18 19:14:21.437625 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.437601    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393f2e3b_3dbb_4d68_b145_ba860b2a0c08.slice/crio-80e64f5a9a1df0f507274cde64470c8640f6f17bd19b998411c763d416ef8f41 WatchSource:0}: Error finding container 80e64f5a9a1df0f507274cde64470c8640f6f17bd19b998411c763d416ef8f41: Status 404 returned error can't find the container with id 80e64f5a9a1df0f507274cde64470c8640f6f17bd19b998411c763d416ef8f41
Sep 18 19:14:21.438405 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.438386    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d130a7b_ffca_47cf_9c74_7e4632c70b4a.slice/crio-b1bd260a390337a021e8455ce0b5462f1fc494028735c220ede9dcebf2d6755a WatchSource:0}: Error finding container b1bd260a390337a021e8455ce0b5462f1fc494028735c220ede9dcebf2d6755a: Status 404 returned error can't find the container with id b1bd260a390337a021e8455ce0b5462f1fc494028735c220ede9dcebf2d6755a
Sep 18 19:14:21.439216 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.439197    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b55544_874b_4f30_baa8_9288cde4cf37.slice/crio-cd4b48cfb31e6d1eabeef77107ed43409759e8c6751ff4404893cd89f463046a WatchSource:0}: Error finding container cd4b48cfb31e6d1eabeef77107ed43409759e8c6751ff4404893cd89f463046a: Status 404 returned error can't find the container with id cd4b48cfb31e6d1eabeef77107ed43409759e8c6751ff4404893cd89f463046a
Sep 18 19:14:21.443736 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:21.443625    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a0cfb3_668d_4396_b407_133871aa0d41.slice/crio-22dbeda0945f75e2a981ba716eb7bd37f7968949c37f1763c4a870b9cfeef27c WatchSource:0}: Error finding container 22dbeda0945f75e2a981ba716eb7bd37f7968949c37f1763c4a870b9cfeef27c: Status 404 returned error can't find the container with id 22dbeda0945f75e2a981ba716eb7bd37f7968949c37f1763c4a870b9cfeef27c
Sep 18 19:14:21.505034 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.505008    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:21.505139 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:21.505105    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:21.516171 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.516144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fs5tv" event={"ID":"7ef57494-7501-4137-8770-70f06d6aa386","Type":"ContainerStarted","Data":"86bfb06eb07e3d07a8b50f201d213ab17cce88c14d754435b2917237aba4bc8d"}
Sep 18 19:14:21.519087 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.519065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl" event={"ID":"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf","Type":"ContainerStarted","Data":"a02fc1900b14aa8888dae57d5a53e36e6eb639f76d6024ee9cef96a4e24e6a94"}
Sep 18 19:14:21.523058 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.523036    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-62z5k" event={"ID":"d2b55544-874b-4f30-baa8-9288cde4cf37","Type":"ContainerStarted","Data":"cd4b48cfb31e6d1eabeef77107ed43409759e8c6751ff4404893cd89f463046a"}
Sep 18 19:14:21.527597 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.527570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"b1bd260a390337a021e8455ce0b5462f1fc494028735c220ede9dcebf2d6755a"}
Sep 18 19:14:21.530601 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.530578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qzkxt" event={"ID":"393f2e3b-3dbb-4d68-b145-ba860b2a0c08","Type":"ContainerStarted","Data":"80e64f5a9a1df0f507274cde64470c8640f6f17bd19b998411c763d416ef8f41"}
Sep 18 19:14:21.532072 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.532052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9rz9" event={"ID":"be4be787-37a0-4434-bfc7-06824039ef7e","Type":"ContainerStarted","Data":"b3bfa62cfa26dfb8bc0e5492080217b5a23c21c728ea3296bda8aede11ceecd6"}
Sep 18 19:14:21.533687 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.533665    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lklth" event={"ID":"f6c26541-181f-4238-9063-4f2fc8fbb422","Type":"ContainerStarted","Data":"35a9a59363ab65bc72ebfc428ccd7485837ac212b4dbbf9157486e9ef1769a06"}
Sep 18 19:14:21.537686 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.537667    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jxxz" event={"ID":"48a0cfb3-668d-4396-b407-133871aa0d41","Type":"ContainerStarted","Data":"22dbeda0945f75e2a981ba716eb7bd37f7968949c37f1763c4a870b9cfeef27c"}
Sep 18 19:14:21.539145 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.539117    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerStarted","Data":"3f7c234ed4ad18b1b2d909a933b71885193cf90bde6ec42c48ea70cc46c2137b"}
Sep 18 19:14:21.542282 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:21.542164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" event={"ID":"bc46e81cf898f71be4cf1f3edbfc59bf","Type":"ContainerStarted","Data":"aa6b5ca6eef04882372d91b9a4068936065523ee7e612a545a57d508de95eb89"}
Sep 18 19:14:22.078963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.078926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:22.079137 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.078978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:22.079137 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.079105    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:22.079236 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.079163    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:24.079145193 +0000 UTC m=+7.797802478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:22.079606 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.079584    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:22.079742 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.079659    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:24.079625825 +0000 UTC m=+7.798283128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:22.179975 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.179942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:22.180121 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.180079    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:22.180121 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.180096    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:22.180121 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.180108    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:22.180266 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.180157    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:24.180139207 +0000 UTC m=+7.898796500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:22.431593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.431455    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:20 +0000 UTC" deadline="2028-05-11 21:57:50.670539025 +0000 UTC"
Sep 18 19:14:22.431593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.431513    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14426h43m28.239030299s"
Sep 18 19:14:22.508711 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.507962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:22.508711 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.508081    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:22.508711 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.508458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:22.508711 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:22.508554    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:22.561586 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.561550    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f03d5f60c7936b409a6408da67cd43c" containerID="76c6f42d6f0cd899a5706ab4590c4c400647a8e5f49105d4a9d744e10ad249fe" exitCode=0
Sep 18 19:14:22.562419 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.562393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"0f03d5f60c7936b409a6408da67cd43c","Type":"ContainerDied","Data":"76c6f42d6f0cd899a5706ab4590c4c400647a8e5f49105d4a9d744e10ad249fe"}
Sep 18 19:14:22.593586 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:22.593346    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-94.ec2.internal" podStartSLOduration=3.5933290749999998 podStartE2EDuration="3.593329075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:21.565109398 +0000 UTC m=+5.283766698" watchObservedRunningTime="2026-09-18 19:14:22.593329075 +0000 UTC m=+6.311986378"
Sep 18 19:14:23.505577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:23.505547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:23.506104 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:23.505690    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:23.578662 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:23.578400    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" event={"ID":"0f03d5f60c7936b409a6408da67cd43c","Type":"ContainerStarted","Data":"737f5cc70d307b01b01a5220d6c5d6efaf1c98d7f78f8ff24ebe67e00bbe4be0"}
Sep 18 19:14:24.093897 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:24.093244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:24.093897 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:24.093296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:24.093897 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.093452    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:24.093897 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.093514    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:28.093496707 +0000 UTC m=+11.812153991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:24.093897 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.093869    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:24.094229 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.093919    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:28.093905339 +0000 UTC m=+11.812562632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:24.195074 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:24.194428    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:24.195074 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.194619    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:24.195074 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.194662    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:24.195074 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.194675    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:24.195074 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.194734    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:28.194716612 +0000 UTC m=+11.913373894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:24.512786 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:24.512759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:24.513298 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:24.512762    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:24.513968 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.513875    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:24.514552 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:24.514517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:25.505083 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:25.505043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:25.505239 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:25.505193    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:26.507516 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:26.507488    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:26.508051 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:26.507599    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:26.508051 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:26.507755    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:26.508051 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:26.507856    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:27.505439 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:27.505408    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:27.505610 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:27.505529    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:28.127542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:28.127594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.127735    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.127792    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:36.127776867 +0000 UTC m=+19.846434170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.128098    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:28.128160 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.128133    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:36.128123598 +0000 UTC m=+19.846780878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:28.228733 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:28.228691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:28.228911 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.228834    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:28.228911 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.228855    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:28.228911 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.228865    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:28.228911 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.228904    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:36.22889181 +0000 UTC m=+19.947549102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:28.506249 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:28.506219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:28.506391 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.506319    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:28.506667 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:28.506648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:28.506750 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:28.506730    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:29.505536 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:29.504967    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:29.505536 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:29.505087    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:30.505426 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:30.505001    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:30.505426 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:30.505037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:30.505426 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:30.505124    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:30.505426 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:30.505207    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:31.505425 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:31.505393    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:31.505582 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:31.505532    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:32.505758 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:32.505725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:32.506182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:32.505725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:32.506182 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:32.505882    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:32.506182 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:32.505916    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:33.505074 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:33.505044    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:33.505257 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:33.505150    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:34.505157 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:34.505121    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:34.505544 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:34.505125    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:34.505544 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:34.505250    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:34.505544 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:34.505351    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:35.505509 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:35.505482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:35.505939 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:35.505592    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:36.183995 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:36.183953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:36.184164 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:36.184002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:36.184164 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.184152    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:36.184263 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.184214    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:52.184193674 +0000 UTC m=+35.902850956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:36.184316 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.184279    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:36.184355 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.184347    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:52.184328142 +0000 UTC m=+35.902985449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:36.284530 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:36.284475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:36.284678 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.284650    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:36.284738 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.284669    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:36.284738 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.284700    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:36.284811 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.284758    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:14:52.284741018 +0000 UTC m=+36.003398314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:36.506346 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:36.505894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:36.506346 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.506011    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:36.506346 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:36.506092    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:36.506346 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:36.506189    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:37.505244 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:37.505211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:37.505419 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:37.505341    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:38.506556 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:38.506522    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:38.506961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:38.506618    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:38.506961 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:38.506660    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:38.506961 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:38.506721    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:39.505359 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.505333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:39.505469 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:39.505446    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:39.616818 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.616565    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jxxz" event={"ID":"48a0cfb3-668d-4396-b407-133871aa0d41","Type":"ContainerStarted","Data":"c7580f0e20c9ef5014051819653a43204a775631e8987a81129205f38c580249"}
Sep 18 19:14:39.618754 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.618706    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerStarted","Data":"6a2f561cb77488d6a5acb09e96007dede85fe592022e0fef5b37c7549c9355ec"}
Sep 18 19:14:39.620756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.620734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fs5tv" event={"ID":"7ef57494-7501-4137-8770-70f06d6aa386","Type":"ContainerStarted","Data":"88621d95aa77e27c88a248b92482a2195d81f547c24a45cb923d488d10a5c8e8"}
Sep 18 19:14:39.622182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.622164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl" event={"ID":"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf","Type":"ContainerStarted","Data":"b1a2e0912454d1009b98952585aa638c1556a1e20b47f20b7e963554ccf4e8b2"}
Sep 18 19:14:39.624100 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.624055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-62z5k" event={"ID":"d2b55544-874b-4f30-baa8-9288cde4cf37","Type":"ContainerStarted","Data":"ba6bf43302ecf6ba3fd1f916af08e31a83a5080d533c7c03d9c569e9f21c3592"}
Sep 18 19:14:39.625786 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.625768    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"237b236e63034adece6b207257b676434fd591785dca7f8c850aca82cc1c5eed"}
Sep 18 19:14:39.627288 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.627267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qzkxt" event={"ID":"393f2e3b-3dbb-4d68-b145-ba860b2a0c08","Type":"ContainerStarted","Data":"8439d9e396a4d4cb6468bbba49606e498ad9ed9aba9f4b97ccf0e7500254ceca"}
Sep 18 19:14:39.629022 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.628761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9rz9" event={"ID":"be4be787-37a0-4434-bfc7-06824039ef7e","Type":"ContainerStarted","Data":"1c4ce45dc7e88e2326765f6a6089a943c0df4557f36b4bc7741c0622f833de89"}
Sep 18 19:14:39.637695 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.637359    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jxxz" podStartSLOduration=3.798822542 podStartE2EDuration="21.637342475s" totalImagesPullingTime="17.838519933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:39.637064359 +0000 UTC m=+23.355721662" watchObservedRunningTime="2026-09-18 19:14:39.637342475 +0000 UTC m=+23.355999777"
Sep 18 19:14:39.637922 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.637798    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-94.ec2.internal" podStartSLOduration=20.63779021 podStartE2EDuration="20.63779021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:23.606056696 +0000 UTC m=+7.324713998" watchObservedRunningTime="2026-09-18 19:14:39.63779021 +0000 UTC m=+23.356447511"
Sep 18 19:14:39.685252 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.685207    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fs5tv" podStartSLOduration=3.938508551 podStartE2EDuration="21.68519099s" totalImagesPullingTime="17.746682439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:39.684812743 +0000 UTC m=+23.403470065" watchObservedRunningTime="2026-09-18 19:14:39.68519099 +0000 UTC m=+23.403848290"
Sep 18 19:14:39.869372 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.869234    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-62z5k" podStartSLOduration=4.120439723 podStartE2EDuration="21.869220649s" totalImagesPullingTime="17.748780926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:39.86671275 +0000 UTC m=+23.585370052" watchObservedRunningTime="2026-09-18 19:14:39.869220649 +0000 UTC m=+23.587877950"
Sep 18 19:14:39.869471 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:39.869451    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qzkxt" podStartSLOduration=4.026001009 podStartE2EDuration="21.869446264s" totalImagesPullingTime="17.843445255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:39.809820796 +0000 UTC m=+23.528478097" watchObservedRunningTime="2026-09-18 19:14:39.869446264 +0000 UTC m=+23.588103565"
Sep 18 19:14:40.505198 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.505170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:40.505399 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:40.505324    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:40.505399 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.505365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:40.505491 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:40.505458    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:40.633577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.633557    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.633847    2584 generic.go:356] "Generic (PLEG): container finished" podID="1d130a7b-ffca-47cf-9c74-7e4632c70b4a" containerID="7928606d48e9d9436142df9dbe4737ea9ded9e69379e9dad500ac3306f173db8" exitCode=1
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.633911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerDied","Data":"7928606d48e9d9436142df9dbe4737ea9ded9e69379e9dad500ac3306f173db8"}
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.633985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"398bc806d2be3a658e6136fc4a1a9c7b70e5040be3e39fbaad0debb68014eb44"}
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.634029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"1977c51e6d2638c33bff98b6ca8263424758a2c656b5044cafa00afda320a04d"}
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.634063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"d1e1211ae1b8890f68793dab882185a1a1be224ab10318cc9d31f7fad81b4266"}
Sep 18 19:14:40.634217 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.634105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"d06511b6f248e4ced9e4b5a73144b886dfac67af6db904f0ea04a977604357d4"}
Sep 18 19:14:40.635326 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.635305    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="6a2f561cb77488d6a5acb09e96007dede85fe592022e0fef5b37c7549c9355ec" exitCode=0
Sep 18 19:14:40.635422 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.635402    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"6a2f561cb77488d6a5acb09e96007dede85fe592022e0fef5b37c7549c9355ec"}
Sep 18 19:14:40.668319 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.668281    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r9rz9" podStartSLOduration=4.862820537 podStartE2EDuration="22.668268174s" totalImagesPullingTime="17.805447637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:39.982286615 +0000 UTC m=+23.700943916" watchObservedRunningTime="2026-09-18 19:14:40.668268174 +0000 UTC m=+24.386925516"
Sep 18 19:14:40.743299 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:40.743281    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:41.444983 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.444857    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:40.743296174Z","UUID":"dc7dd1b2-16eb-4495-b1f1-05624cd08085","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:41.446577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.446551    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:41.446718 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.446593    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:41.505651 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.505610    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:41.505797 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:41.505759    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:41.642560 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.640843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl" event={"ID":"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf","Type":"ContainerStarted","Data":"3b6bbdb4350d75d6797e27e315af15762e3458028f33963355953644c029f291"}
Sep 18 19:14:41.642560 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:41.642503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lklth" event={"ID":"f6c26541-181f-4238-9063-4f2fc8fbb422","Type":"ContainerStarted","Data":"522040a6124b41963b5455646beb287da9909e9eecfc3881d144a30d1afaf2a6"}
Sep 18 19:14:42.505305 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.505277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:42.505542 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.505292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:42.505542 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:42.505412    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:42.505542 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:42.505462    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:42.647256 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.647218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl" event={"ID":"36feb807-ad9b-47bd-a2c9-a1b3cf6adbaf","Type":"ContainerStarted","Data":"809b5216b69d52a4005e74a20be92afd2dc57261969fff34213b900e8575c1ea"}
Sep 18 19:14:42.651619 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.651596    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:14:42.652049 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.652024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"a077da7ba58b3b80936ff04027a91c1cb80aeedbb3a7aa7288f6d5e15a1fa99d"}
Sep 18 19:14:42.680765 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.680717    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvdl" podStartSLOduration=4.924200452 podStartE2EDuration="24.680700741s" totalImagesPullingTime="19.756500289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:42.680537546 +0000 UTC m=+26.399194847" watchObservedRunningTime="2026-09-18 19:14:42.680700741 +0000 UTC m=+26.399358043"
Sep 18 19:14:42.681013 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:42.680983    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lklth" podStartSLOduration=6.836327115 podStartE2EDuration="24.680970923s" totalImagesPullingTime="17.844643808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:41.673365361 +0000 UTC m=+25.392022662" watchObservedRunningTime="2026-09-18 19:14:42.680970923 +0000 UTC m=+26.399628226"
Sep 18 19:14:43.505266 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:43.505236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:43.505401 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:43.505368    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:44.506007 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.505966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:44.506007 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.506007    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:44.506480 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:44.506109    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:44.506480 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:44.506219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:44.571865 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.571835    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:44.572463 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.572442    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:44.657226 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.657200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:44.657619 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:44.657601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fs5tv"
Sep 18 19:14:45.505970 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.505805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:45.506088 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:45.506066    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:45.661652 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.661613    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="689e14c3c81874382c6e57470b2532e4cc980648d9d456da3a24e87cb9b3c421" exitCode=0
Sep 18 19:14:45.661765 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.661741    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"689e14c3c81874382c6e57470b2532e4cc980648d9d456da3a24e87cb9b3c421"}
Sep 18 19:14:45.666407 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.666381    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:14:45.666925 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.666893    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"2b430c510667f9ed3e95c32a3416988b6b63c399ed7ddde4e5a0b9375a10d761"}
Sep 18 19:14:45.667213 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.667195    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:45.667410 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.667392    2584 scope.go:117] "RemoveContainer" containerID="7928606d48e9d9436142df9dbe4737ea9ded9e69379e9dad500ac3306f173db8"
Sep 18 19:14:45.684064 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.684042    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:45.812793 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:45.812744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:46.509152 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.508962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:46.509608 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.509004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:46.509608 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:46.509276    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:46.509608 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:46.509386    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:46.676383 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.676363    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:14:46.676851 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.676824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" event={"ID":"1d130a7b-ffca-47cf-9c74-7e4632c70b4a","Type":"ContainerStarted","Data":"4309bde5f57b534223c9893d88baa240b4567fa950ea8a50be1634169a7ecea9"}
Sep 18 19:14:46.677125 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.677101    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:46.679213 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.679191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerStarted","Data":"e5e76395560d74da0afe72b2aebb9d30edd1ba0a99706332d0073b7b0b4c2fd6"}
Sep 18 19:14:46.694116 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.694095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:14:46.722428 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.722390    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c" podStartSLOduration=10.884681947 podStartE2EDuration="28.722379232s" totalImagesPullingTime="17.837697285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:46.721172921 +0000 UTC m=+30.439830221" watchObservedRunningTime="2026-09-18 19:14:46.722379232 +0000 UTC m=+30.441036532"
Sep 18 19:14:46.770771 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.770696    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7thst"]
Sep 18 19:14:46.770852 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.770772    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:46.770852 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:46.770847    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:46.774108 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.773652    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6td6"]
Sep 18 19:14:46.774108 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.773753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:46.774108 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:46.773838    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:46.774302 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.774196    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9829"]
Sep 18 19:14:46.774302 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:46.774280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:46.774431 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:46.774367    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:47.683716 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:47.683680    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="e5e76395560d74da0afe72b2aebb9d30edd1ba0a99706332d0073b7b0b4c2fd6" exitCode=0
Sep 18 19:14:47.684239 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:47.683763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"e5e76395560d74da0afe72b2aebb9d30edd1ba0a99706332d0073b7b0b4c2fd6"}
Sep 18 19:14:48.505894 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:48.505863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:48.506045 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:48.505954    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:48.506045 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:48.506033    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:48.506148 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:48.506114    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:48.506148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:48.506141    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:48.506251 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:48.506218    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:49.690820 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:49.690791    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="079f587bcdd819adc88bbf32ca594029c8cf5f2dd1194bf0ca7b4ab21af0e378" exitCode=0
Sep 18 19:14:49.691185 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:49.690826    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"079f587bcdd819adc88bbf32ca594029c8cf5f2dd1194bf0ca7b4ab21af0e378"}
Sep 18 19:14:50.509424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:50.509394    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:50.509595 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:50.509509    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6td6" podUID="eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe"
Sep 18 19:14:50.509595 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:50.509517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:50.509699 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:50.509520    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:50.509699 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:50.509602    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7thst" podUID="f6e38969-99fd-4925-9313-6ef4ac0674c2"
Sep 18 19:14:50.509817 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:50.509742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:14:52.116630 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.116411    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-94.ec2.internal" event="NodeReady"
Sep 18 19:14:52.116997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.116780    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:14:52.193969 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.193904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:52.193969 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.193944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:52.194151 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.194038    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:52.194151 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.194042    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:52.194151 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.194097    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.194078103 +0000 UTC m=+67.912735386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:52.194151 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.194116    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret podName:f6e38969-99fd-4925-9313-6ef4ac0674c2 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.194105671 +0000 UTC m=+67.912762953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret") pod "global-pull-secret-syncer-7thst" (UID: "f6e38969-99fd-4925-9313-6ef4ac0674c2") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:52.234709 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.234685    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-flnrv"]
Sep 18 19:14:52.266666 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.266626    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pfs8t"]
Sep 18 19:14:52.266795 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.266750    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.270304 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.270281    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:14:52.277405 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.277385    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7n8q\""
Sep 18 19:14:52.277405 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.277397    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:14:52.280440 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.280417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pfs8t"]
Sep 18 19:14:52.280532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.280456    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flnrv"]
Sep 18 19:14:52.280578 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.280541    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.292225 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.292197    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:14:52.292338 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.292253    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:14:52.292338 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.292329    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:14:52.292728 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.292709    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lcfbs\""
Sep 18 19:14:52.294895 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.294871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:52.295044 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.295028    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:52.295094 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.295047    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:52.295094 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.295057    2584 projected.go:196] Error preparing data for projected volume kube-api-access-l28d5 for pod openshift-network-diagnostics/network-check-target-c6td6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:52.295178 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.295116    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5 podName:eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.295099382 +0000 UTC m=+68.013756683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l28d5" (UniqueName: "kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5") pod "network-check-target-c6td6" (UID: "eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:52.396088 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396061    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09842818-d867-4448-b34b-cdded954cb09-cert\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.396228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.396228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396146    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df114d-0390-4d42-b107-656830b8a27c-tmp-dir\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.396228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rnw\" (UniqueName: \"kubernetes.io/projected/64df114d-0390-4d42-b107-656830b8a27c-kube-api-access-p5rnw\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.396228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/09842818-d867-4448-b34b-cdded954cb09-kube-api-access-rb5df\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.396435 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.396331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64df114d-0390-4d42-b107-656830b8a27c-config-volume\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.497581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.497581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df114d-0390-4d42-b107-656830b8a27c-tmp-dir\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.497581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rnw\" (UniqueName: \"kubernetes.io/projected/64df114d-0390-4d42-b107-656830b8a27c-kube-api-access-p5rnw\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.497836 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497592    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/09842818-d867-4448-b34b-cdded954cb09-kube-api-access-rb5df\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.497836 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64df114d-0390-4d42-b107-656830b8a27c-config-volume\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.497836 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.497694    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:52.497836 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.497714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09842818-d867-4448-b34b-cdded954cb09-cert\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.497836 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:52.497753    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:52.997735641 +0000 UTC m=+36.716392943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:52.498160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.498140    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64df114d-0390-4d42-b107-656830b8a27c-tmp-dir\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.498405 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.498379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/64df114d-0390-4d42-b107-656830b8a27c-config-volume\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.502480 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.502457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09842818-d867-4448-b34b-cdded954cb09-cert\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.508964 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.508943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:14:52.509057 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.509043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:14:52.509114 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.509066    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:14:52.513897 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.513879    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:14:52.513997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.513912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:14:52.513997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.513990    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:14:52.514158 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.514139    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:14:52.514545 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.514527    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w2qlj\""
Sep 18 19:14:52.514705 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.514691    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5n7ds\""
Sep 18 19:14:52.527136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.527114    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/09842818-d867-4448-b34b-cdded954cb09-kube-api-access-rb5df\") pod \"ingress-canary-pfs8t\" (UID: \"09842818-d867-4448-b34b-cdded954cb09\") " pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:52.527212 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.527163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rnw\" (UniqueName: \"kubernetes.io/projected/64df114d-0390-4d42-b107-656830b8a27c-kube-api-access-p5rnw\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:52.590823 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:52.590804    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pfs8t"
Sep 18 19:14:53.001194 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:53.001153    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:53.001366 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:53.001330    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:53.001420 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:53.001412    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:54.001390819 +0000 UTC m=+37.720048104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:53.890944 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:53.890888    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pfs8t"]
Sep 18 19:14:53.901372 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:14:53.901340    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09842818_d867_4448_b34b_cdded954cb09.slice/crio-cbf84bb1da152d2bec056b721fdf4220fd03474e68a62902554024a6a0d30269 WatchSource:0}: Error finding container cbf84bb1da152d2bec056b721fdf4220fd03474e68a62902554024a6a0d30269: Status 404 returned error can't find the container with id cbf84bb1da152d2bec056b721fdf4220fd03474e68a62902554024a6a0d30269
Sep 18 19:14:54.008524 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:54.008490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:54.008686 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:54.008666    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:54.008739 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:54.008733    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:14:56.008715234 +0000 UTC m=+39.727372516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:54.705325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:54.705075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pfs8t" event={"ID":"09842818-d867-4448-b34b-cdded954cb09","Type":"ContainerStarted","Data":"cbf84bb1da152d2bec056b721fdf4220fd03474e68a62902554024a6a0d30269"}
Sep 18 19:14:54.708495 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:54.708466    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="fd801410057237b3fb82f61abea5d70010477df98773f70dac14efd5a90821b0" exitCode=0
Sep 18 19:14:54.708608 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:54.708503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"fd801410057237b3fb82f61abea5d70010477df98773f70dac14efd5a90821b0"}
Sep 18 19:14:55.714948 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:55.714912    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f46f69a-eedc-4d12-9a53-96baae2780b6" containerID="986510fad0d544227e7158deeaa570701f251a33af427a07124ab6ab712b24f9" exitCode=0
Sep 18 19:14:55.715377 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:55.714955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerDied","Data":"986510fad0d544227e7158deeaa570701f251a33af427a07124ab6ab712b24f9"}
Sep 18 19:14:56.020505 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:56.020473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:14:56.020701 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:56.020681    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:14:56.020769 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:14:56.020747    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:00.020726892 +0000 UTC m=+43.739384172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:14:56.721314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:56.721230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" event={"ID":"0f46f69a-eedc-4d12-9a53-96baae2780b6","Type":"ContainerStarted","Data":"07f5a97d643ec58cd1c09ab8c8e1f6e0f90d31dd0bcf742de20b1635b2333d3b"}
Sep 18 19:14:56.722764 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:56.722740    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pfs8t" event={"ID":"09842818-d867-4448-b34b-cdded954cb09","Type":"ContainerStarted","Data":"274a8c6f424ad39fce4bc2b55e1c9ff80624a57862cfe321f3cea8f60c9d8b2e"}
Sep 18 19:14:56.812862 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:56.812816    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fsx2n" podStartSLOduration=10.628086025 podStartE2EDuration="38.812802065s" totalImagesPullingTime="28.18471604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:56.781888652 +0000 UTC m=+40.500545953" watchObservedRunningTime="2026-09-18 19:14:56.812802065 +0000 UTC m=+40.531459357"
Sep 18 19:14:56.870558 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:14:56.870518    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pfs8t" podStartSLOduration=2.747910516 podStartE2EDuration="4.870504418s" totalImagesPullingTime="2.122593902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:56.870217393 +0000 UTC m=+40.588874694" watchObservedRunningTime="2026-09-18 19:14:56.870504418 +0000 UTC m=+40.589161757"
Sep 18 19:15:00.048000 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:00.047949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:15:00.048455 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:00.048099    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:00.048455 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:00.048168    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:08.048151022 +0000 UTC m=+51.766808301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:08.100830 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:08.100790    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:15:08.101261 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:08.100931    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:08.101261 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:08.101003    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:24.100986493 +0000 UTC m=+67.819643796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:18.697364 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:18.697334    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5x2c"
Sep 18 19:15:24.104406 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.104354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:15:24.104833 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:24.104502    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:24.104833 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:24.104580    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:15:56.10456314 +0000 UTC m=+99.823220419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:24.205286 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.205251    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:15:24.205286 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.205290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:15:24.208525 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.208490    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:24.208695 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.208679    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:24.216095 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:24.216080    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:15:24.216161 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:24.216147    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:28.216133314 +0000 UTC m=+131.934790593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : secret "metrics-daemon-secret" not found
Sep 18 19:15:24.219372 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.219355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6e38969-99fd-4925-9313-6ef4ac0674c2-original-pull-secret\") pod \"global-pull-secret-syncer-7thst\" (UID: \"f6e38969-99fd-4925-9313-6ef4ac0674c2\") " pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:15:24.306329 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.306291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:15:24.308981 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.308965    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:24.319315 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.319293    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:24.320331 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.320318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7thst"
Sep 18 19:15:24.330819 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.330799    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28d5\" (UniqueName: \"kubernetes.io/projected/eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe-kube-api-access-l28d5\") pod \"network-check-target-c6td6\" (UID: \"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe\") " pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:15:24.604461 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.604429    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7thst"]
Sep 18 19:15:24.609233 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:15:24.609206    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e38969_99fd_4925_9313_6ef4ac0674c2.slice/crio-b5008792e890b47a6da5d72ca04fa98790c2610396d6a6202d786257c9d36d81 WatchSource:0}: Error finding container b5008792e890b47a6da5d72ca04fa98790c2610396d6a6202d786257c9d36d81: Status 404 returned error can't find the container with id b5008792e890b47a6da5d72ca04fa98790c2610396d6a6202d786257c9d36d81
Sep 18 19:15:24.629849 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.629794    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5n7ds\""
Sep 18 19:15:24.637993 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.637975    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:15:24.790362 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.790319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7thst" event={"ID":"f6e38969-99fd-4925-9313-6ef4ac0674c2","Type":"ContainerStarted","Data":"b5008792e890b47a6da5d72ca04fa98790c2610396d6a6202d786257c9d36d81"}
Sep 18 19:15:24.910309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:24.910231    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6td6"]
Sep 18 19:15:24.913573 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:15:24.913550    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0f6e0b_9fb2_4d68_9f73_494449ba0bbe.slice/crio-dc22904728b13fff03032e5d72551dab57900b4db0ce8876522a856661267740 WatchSource:0}: Error finding container dc22904728b13fff03032e5d72551dab57900b4db0ce8876522a856661267740: Status 404 returned error can't find the container with id dc22904728b13fff03032e5d72551dab57900b4db0ce8876522a856661267740
Sep 18 19:15:25.794743 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:25.794703    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6td6" event={"ID":"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe","Type":"ContainerStarted","Data":"dc22904728b13fff03032e5d72551dab57900b4db0ce8876522a856661267740"}
Sep 18 19:15:29.807656 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:29.807602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6td6" event={"ID":"eb0f6e0b-9fb2-4d68-9f73-494449ba0bbe","Type":"ContainerStarted","Data":"ea20d907cefc2d6b345b7702333f939d20412645928dfb66df11b74758a1f0f8"}
Sep 18 19:15:29.808094 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:29.807680    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:15:29.809117 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:29.809092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7thst" event={"ID":"f6e38969-99fd-4925-9313-6ef4ac0674c2","Type":"ContainerStarted","Data":"62bfcae2633c97986febecf1bb29b1fb157af888e6aaf322a042f0a36c366b4e"}
Sep 18 19:15:29.831183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:29.831132    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c6td6" podStartSLOduration=67.570583642 podStartE2EDuration="1m11.831115742s" totalImagesPullingTime="4.2605321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:29.830414316 +0000 UTC m=+73.549071618" watchObservedRunningTime="2026-09-18 19:15:29.831115742 +0000 UTC m=+73.549773075"
Sep 18 19:15:29.847286 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:29.847227    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7thst" podStartSLOduration=67.282142233 podStartE2EDuration="1m11.84720838s" totalImagesPullingTime="4.565066147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:29.84697339 +0000 UTC m=+73.565630692" watchObservedRunningTime="2026-09-18 19:15:29.84720838 +0000 UTC m=+73.565865697"
Sep 18 19:15:56.113691 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:15:56.113631    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:15:56.114180 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:56.113737    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:56.114180 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:15:56.113803    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls podName:64df114d-0390-4d42-b107-656830b8a27c nodeName:}" failed. No retries permitted until 2026-09-18 19:17:00.113787147 +0000 UTC m=+163.832444426 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls") pod "dns-default-flnrv" (UID: "64df114d-0390-4d42-b107-656830b8a27c") : secret "dns-default-metrics-tls" not found
Sep 18 19:16:00.814777 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:00.814737    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c6td6"
Sep 18 19:16:09.469772 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.469734    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"]
Sep 18 19:16:09.472418 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.472395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.482969 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.482950    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:16:09.485028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.485009    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:16:09.485105 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.485045    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:16:09.485105 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.485015    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:16:09.485105 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.485009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9c9h4\""
Sep 18 19:16:09.497938 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.497916    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"]
Sep 18 19:16:09.503922 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.503902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58a99a04-199f-409c-80c5-ecf84c2ba538-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.504005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.503936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74r6w\" (UniqueName: \"kubernetes.io/projected/58a99a04-199f-409c-80c5-ecf84c2ba538-kube-api-access-74r6w\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.504005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.503975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.605010 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.604986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58a99a04-199f-409c-80c5-ecf84c2ba538-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.605120 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.605019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74r6w\" (UniqueName: \"kubernetes.io/projected/58a99a04-199f-409c-80c5-ecf84c2ba538-kube-api-access-74r6w\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.605120 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.605047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.605196 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:09.605136    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:09.605196 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:09.605192    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:10.105175611 +0000 UTC m=+113.823832907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:09.605742 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.605725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/58a99a04-199f-409c-80c5-ecf84c2ba538-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.627438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.627411    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74r6w\" (UniqueName: \"kubernetes.io/projected/58a99a04-199f-409c-80c5-ecf84c2ba538-kube-api-access-74r6w\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:09.696651 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.696615    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-chdrd"]
Sep 18 19:16:09.698628 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.698614    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.702241 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.702221    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:16:09.702351 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.702291    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:16:09.703011 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.702994    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:16:09.703106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.703083    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kxthq\""
Sep 18 19:16:09.703204 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.703188    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:16:09.710067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.710046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:16:09.716647 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.716610    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-chdrd"]
Sep 18 19:16:09.793035 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.793005    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"]
Sep 18 19:16:09.794750 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.794735    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"
Sep 18 19:16:09.797577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.797560    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvjq6\""
Sep 18 19:16:09.797895 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.797883    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"]
Sep 18 19:16:09.799504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.799478    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"]
Sep 18 19:16:09.799601 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.799583    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:09.801227 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.801212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.803980 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.803959    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:16:09.805798 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpkk\" (UniqueName: \"kubernetes.io/projected/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-kube-api-access-rjpkk\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.805892 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-snapshots\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.805892 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.805892 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805842    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.806058 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-tmp\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.806058 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.805938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-serving-cert\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.806250 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806229    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:16:09.806347 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806290    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:16:09.806417 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806345    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:09.806417 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806236    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:09.806631 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806612    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:16:09.806631 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806629    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:16:09.806957 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.806945    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:16:09.807231 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.807218    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:16:09.815711 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.815693    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"]
Sep 18 19:16:09.826746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.826720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s8ggh\""
Sep 18 19:16:09.826856 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.826810    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:16:09.827685 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.827668    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"]
Sep 18 19:16:09.883028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.882997    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"]
Sep 18 19:16:09.907160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpkk\" (UniqueName: \"kubernetes.io/projected/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-kube-api-access-rjpkk\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907311 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907170    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.907311 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:09.907438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-snapshots\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khnj\" (UniqueName: \"kubernetes.io/projected/e859d994-b24c-429d-b336-a784dde75210-kube-api-access-8khnj\") pod \"network-check-source-7cbc45fddc-lcmjp\" (UID: \"e859d994-b24c-429d-b336-a784dde75210\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"
Sep 18 19:16:09.907438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a4726-8924-45c7-aea8-bb203ee53c6f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2vs\" (UniqueName: \"kubernetes.io/projected/fdb25206-676e-4a86-9bb1-048942d9b8f3-kube-api-access-ln2vs\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907538    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-tmp\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907610    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-serving-cert\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.907746 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.907664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgq5r\" (UniqueName: \"kubernetes.io/projected/fe0a4726-8924-45c7-aea8-bb203ee53c6f-kube-api-access-zgq5r\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:09.908085 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.908065    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.908137 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.908099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-snapshots\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.908192 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.908151    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-tmp\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.908273 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.908256    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.910232 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.910211    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-serving-cert\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:09.954276 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:09.954251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpkk\" (UniqueName: \"kubernetes.io/projected/2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8-kube-api-access-rjpkk\") pod \"insights-operator-6f9c644dd9-chdrd\" (UID: \"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8\") " pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:10.008247 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd"
Sep 18 19:16:10.008410 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008484 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008484 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgq5r\" (UniqueName: \"kubernetes.io/projected/fe0a4726-8924-45c7-aea8-bb203ee53c6f-kube-api-access-zgq5r\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008588 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008588 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:10.008730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khnj\" (UniqueName: \"kubernetes.io/projected/e859d994-b24c-429d-b336-a784dde75210-kube-api-access-8khnj\") pod \"network-check-source-7cbc45fddc-lcmjp\" (UID: \"e859d994-b24c-429d-b336-a784dde75210\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"
Sep 18 19:16:10.008730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a4726-8924-45c7-aea8-bb203ee53c6f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2vs\" (UniqueName: \"kubernetes.io/projected/fdb25206-676e-4a86-9bb1-048942d9b8f3-kube-api-access-ln2vs\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:10.008873 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.008735    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.008929 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.008712    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:10.009020 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.008996    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls podName:fdb25206-676e-4a86-9bb1-048942d9b8f3 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:10.508968123 +0000 UTC m=+114.227625403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b2dpm" (UID: "fdb25206-676e-4a86-9bb1-048942d9b8f3") : secret "samples-operator-tls" not found
Sep 18 19:16:10.009752 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.009719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a4726-8924-45c7-aea8-bb203ee53c6f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.011168 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.011130    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.011210 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.011172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.011299 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.011280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.011876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.011856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fe0a4726-8924-45c7-aea8-bb203ee53c6f-ca\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.067099 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.067051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgq5r\" (UniqueName: \"kubernetes.io/projected/fe0a4726-8924-45c7-aea8-bb203ee53c6f-kube-api-access-zgq5r\") pod \"cluster-proxy-proxy-agent-ffcf79c45-2ftk2\" (UID: \"fe0a4726-8924-45c7-aea8-bb203ee53c6f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.109783 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.109758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:10.109930 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.109912    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:10.109992 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.109982    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:11.109963012 +0000 UTC m=+114.828620293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:10.127869 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.127844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:16:10.128895 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.128878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khnj\" (UniqueName: \"kubernetes.io/projected/e859d994-b24c-429d-b336-a784dde75210-kube-api-access-8khnj\") pod \"network-check-source-7cbc45fddc-lcmjp\" (UID: \"e859d994-b24c-429d-b336-a784dde75210\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"
Sep 18 19:16:10.133337 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.133320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2vs\" (UniqueName: \"kubernetes.io/projected/fdb25206-676e-4a86-9bb1-048942d9b8f3-kube-api-access-ln2vs\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:10.296490 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.296467    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-chdrd"]
Sep 18 19:16:10.298961 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:10.298935    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2a79a7_d8c7_4dd3_b0b7_80e4add24cd8.slice/crio-174fc484f558d359c36601b6433abbc613387c10cd9585352ce8a7fbc3ecef44 WatchSource:0}: Error finding container 174fc484f558d359c36601b6433abbc613387c10cd9585352ce8a7fbc3ecef44: Status 404 returned error can't find the container with id 174fc484f558d359c36601b6433abbc613387c10cd9585352ce8a7fbc3ecef44
Sep 18 19:16:10.404988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.404959    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"
Sep 18 19:16:10.408021 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.408000    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"]
Sep 18 19:16:10.411494 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:10.411472    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe0a4726_8924_45c7_aea8_bb203ee53c6f.slice/crio-5cc877c9fb12db9f4280d4f2c70ef9e203b54615d3dfad5b59ec45211a493f4b WatchSource:0}: Error finding container 5cc877c9fb12db9f4280d4f2c70ef9e203b54615d3dfad5b59ec45211a493f4b: Status 404 returned error can't find the container with id 5cc877c9fb12db9f4280d4f2c70ef9e203b54615d3dfad5b59ec45211a493f4b
Sep 18 19:16:10.512334 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.512307    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:10.512617 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.512444    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:10.512617 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:10.512499    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls podName:fdb25206-676e-4a86-9bb1-048942d9b8f3 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:11.512485346 +0000 UTC m=+115.231142626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b2dpm" (UID: "fdb25206-676e-4a86-9bb1-048942d9b8f3") : secret "samples-operator-tls" not found
Sep 18 19:16:10.678512 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.678450    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp"]
Sep 18 19:16:10.681736 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:10.681706    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode859d994_b24c_429d_b336_a784dde75210.slice/crio-54b99462ba0b00940ac3bb41167f7af1d5c9d227400ee160372ec48753f25c6f WatchSource:0}: Error finding container 54b99462ba0b00940ac3bb41167f7af1d5c9d227400ee160372ec48753f25c6f: Status 404 returned error can't find the container with id 54b99462ba0b00940ac3bb41167f7af1d5c9d227400ee160372ec48753f25c6f
Sep 18 19:16:10.914156 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.914117    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp" event={"ID":"e859d994-b24c-429d-b336-a784dde75210","Type":"ContainerStarted","Data":"6880b2759968c81c1fdc71269894ef7caf8e1c65ceb53ad82000a9a9c75fb522"}
Sep 18 19:16:10.914345 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.914179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp" event={"ID":"e859d994-b24c-429d-b336-a784dde75210","Type":"ContainerStarted","Data":"54b99462ba0b00940ac3bb41167f7af1d5c9d227400ee160372ec48753f25c6f"}
Sep 18 19:16:10.915779 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.915750    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd" event={"ID":"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8","Type":"ContainerStarted","Data":"174fc484f558d359c36601b6433abbc613387c10cd9585352ce8a7fbc3ecef44"}
Sep 18 19:16:10.917094 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.917072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerStarted","Data":"5cc877c9fb12db9f4280d4f2c70ef9e203b54615d3dfad5b59ec45211a493f4b"}
Sep 18 19:16:10.935085 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:10.932769    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-lcmjp" podStartSLOduration=1.932751385 podStartE2EDuration="1.932751385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:10.930710776 +0000 UTC m=+114.649368077" watchObservedRunningTime="2026-09-18 19:16:10.932751385 +0000 UTC m=+114.651408700"
Sep 18 19:16:11.116879 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:11.116839    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:11.117078 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:11.117058    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:11.117215 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:11.117140    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:13.117123285 +0000 UTC m=+116.835780563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:11.520105 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:11.520060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:11.520569 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:11.520253    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:11.520569 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:11.520329    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls podName:fdb25206-676e-4a86-9bb1-048942d9b8f3 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:13.520310326 +0000 UTC m=+117.238967612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b2dpm" (UID: "fdb25206-676e-4a86-9bb1-048942d9b8f3") : secret "samples-operator-tls" not found
Sep 18 19:16:13.133969 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:13.133929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:13.134347 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:13.134069    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:13.134347 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:13.134145    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:17.134128484 +0000 UTC m=+120.852785783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:13.537473 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:13.537436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:13.537666 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:13.537593    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:13.537749 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:13.537690    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls podName:fdb25206-676e-4a86-9bb1-048942d9b8f3 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:17.537668025 +0000 UTC m=+121.256325304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b2dpm" (UID: "fdb25206-676e-4a86-9bb1-048942d9b8f3") : secret "samples-operator-tls" not found
Sep 18 19:16:14.931658 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:14.931598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd" event={"ID":"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8","Type":"ContainerStarted","Data":"d8199707ee9e7acb320f9f33d4d670f75fc53b03ad63579f24e448371f79acb7"}
Sep 18 19:16:14.933764 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:14.933730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerStarted","Data":"b680e935f9bf2da3b13208dea87de3f2aa4a232f0bf54d9827fb9253682d4c6c"}
Sep 18 19:16:14.933883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:14.933771    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerStarted","Data":"85d318740bc6fbad9ef615d96b0271f931e9027284eff9beca12524e2a8c0abf"}
Sep 18 19:16:14.933883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:14.933785    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerStarted","Data":"678d1a0804fd2c31c853a5e54b2141479974012ddc578e36939211afa7bbc587"}
Sep 18 19:16:15.067950 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.067901    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" podStartSLOduration=2.271975091 podStartE2EDuration="6.067881445s" totalImagesPullingTime="3.795906354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:15.067840396 +0000 UTC m=+118.786497698" watchObservedRunningTime="2026-09-18 19:16:15.067881445 +0000 UTC m=+118.786538748"
Sep 18 19:16:15.068285 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.068266    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd" podStartSLOduration=2.193920092 podStartE2EDuration="6.068260625s" totalImagesPullingTime="3.874340533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:14.997874676 +0000 UTC m=+118.716531977" watchObservedRunningTime="2026-09-18 19:16:15.068260625 +0000 UTC m=+118.786917930"
Sep 18 19:16:15.487565 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.487521    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rbnz8"]
Sep 18 19:16:15.489684 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.489669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.499338 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.499320    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:16:15.499909 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.499890    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-npv6k\""
Sep 18 19:16:15.500161 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.500141    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:16:15.514463 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.514437    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rbnz8"]
Sep 18 19:16:15.553717 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.553685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e13f0853-dab1-403c-972f-0079739bd48a-crio-socket\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.553844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.553725    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmhb\" (UniqueName: \"kubernetes.io/projected/e13f0853-dab1-403c-972f-0079739bd48a-kube-api-access-kfmhb\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.553844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.553772    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.553844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.553790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.553844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.553828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e13f0853-dab1-403c-972f-0079739bd48a-data-volume\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654704 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654704 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e13f0853-dab1-403c-972f-0079739bd48a-data-volume\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e13f0853-dab1-403c-972f-0079739bd48a-crio-socket\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654833    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmhb\" (UniqueName: \"kubernetes.io/projected/e13f0853-dab1-403c-972f-0079739bd48a-kube-api-access-kfmhb\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.654901 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:15.654857    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:15.654901 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:15.654862    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:15.655098 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.654920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e13f0853-dab1-403c-972f-0079739bd48a-crio-socket\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.655098 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:15.654929    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:16.154910299 +0000 UTC m=+119.873567601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:15.655098 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:15.654985    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:16.154964171 +0000 UTC m=+119.873621453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:15.655196 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.655102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e13f0853-dab1-403c-972f-0079739bd48a-data-volume\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:15.667207 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:15.667182    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmhb\" (UniqueName: \"kubernetes.io/projected/e13f0853-dab1-403c-972f-0079739bd48a-kube-api-access-kfmhb\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:16.159323 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:16.159286    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:16.159323 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:16.159323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:16.159759 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:16.159424    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:16.159759 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:16.159462    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:16.159759 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:16.159487    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:17.15947208 +0000 UTC m=+120.878129364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:16.159759 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:16.159516    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:17.159503497 +0000 UTC m=+120.878160795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:17.090017 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:17.089990    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jxxz_48a0cfb3-668d-4396-b407-133871aa0d41/dns-node-resolver/0.log"
Sep 18 19:16:17.166472 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:17.166444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:17.166539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:17.166559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166594    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166624    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166691    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:25.166672848 +0000 UTC m=+128.885330143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166692    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166711    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:19.166701398 +0000 UTC m=+122.885358679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:17.166837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.166743    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:19.166733725 +0000 UTC m=+122.885391021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:17.570057 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:17.570029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:17.570191 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.570171    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:17.570249 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:17.570240    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls podName:fdb25206-676e-4a86-9bb1-048942d9b8f3 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:25.570224421 +0000 UTC m=+129.288881700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-b2dpm" (UID: "fdb25206-676e-4a86-9bb1-048942d9b8f3") : secret "samples-operator-tls" not found
Sep 18 19:16:18.038017 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:18.037992    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-62z5k_d2b55544-874b-4f30-baa8-9288cde4cf37/node-ca/0.log"
Sep 18 19:16:19.179914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:19.179882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:19.179914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:19.179918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:19.180300 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:19.179979    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:19.180300 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:19.180030    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:23.180017663 +0000 UTC m=+126.898674956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:19.180300 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:19.180030    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:19.180300 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:19.180079    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:23.180066281 +0000 UTC m=+126.898723578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:19.260392 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:19.260360    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pfs8t_09842818-d867-4448-b34b-cdded954cb09/serve-healthcheck-canary/0.log"
Sep 18 19:16:23.210071 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:23.210039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:23.210071 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:23.210071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:23.210531 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:23.210145    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:23.210531 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:23.210203    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.210187355 +0000 UTC m=+134.928844640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:23.210531 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:23.210211    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:23.210531 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:23.210279    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.210261311 +0000 UTC m=+134.928918595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:25.226399 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.226372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:25.226823 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:25.226491    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:25.226823 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:25.226541    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls podName:58a99a04-199f-409c-80c5-ecf84c2ba538 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:41.226528212 +0000 UTC m=+144.945185491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vrnxw" (UID: "58a99a04-199f-409c-80c5-ecf84c2ba538") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:25.630851 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.630815    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:25.633318 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.633288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdb25206-676e-4a86-9bb1-048942d9b8f3-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-b2dpm\" (UID: \"fdb25206-676e-4a86-9bb1-048942d9b8f3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:25.713454 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.713429    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s8ggh\""
Sep 18 19:16:25.722175 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.722160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"
Sep 18 19:16:25.993658 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:25.993567    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm"]
Sep 18 19:16:26.969361 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:26.969322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm" event={"ID":"fdb25206-676e-4a86-9bb1-048942d9b8f3","Type":"ContainerStarted","Data":"e25c47afe01dfa99ce19d7dc2397ca7683f9914952cbe51ec3777176b094359f"}
Sep 18 19:16:27.974384 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:27.974296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm" event={"ID":"fdb25206-676e-4a86-9bb1-048942d9b8f3","Type":"ContainerStarted","Data":"78bf88ea1361afcdc963d37f33eb5daf16f21e2b97b12e77693a048714dec863"}
Sep 18 19:16:27.974384 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:27.974343    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm" event={"ID":"fdb25206-676e-4a86-9bb1-048942d9b8f3","Type":"ContainerStarted","Data":"937a0b44176979a2aa0cbef31c5a680b29fefbcbf3c184ca77e78e55c642f919"}
Sep 18 19:16:27.991581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:27.991520    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-b2dpm" podStartSLOduration=17.465067887 podStartE2EDuration="18.99150354s" totalImagesPullingTime="1.526435653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:27.990924033 +0000 UTC m=+131.709581334" watchObservedRunningTime="2026-09-18 19:16:27.99150354 +0000 UTC m=+131.710160841"
Sep 18 19:16:28.252306 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:28.252276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:16:28.252461 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:28.252422    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:16:28.252516 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:28.252485    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs podName:738fc24a-cf2d-4ccc-995e-eb9e54e6b044 nodeName:}" failed. No retries permitted until 2026-09-18 19:18:30.252469317 +0000 UTC m=+253.971126616 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs") pod "network-metrics-daemon-l9829" (UID: "738fc24a-cf2d-4ccc-995e-eb9e54e6b044") : secret "metrics-daemon-secret" not found
Sep 18 19:16:31.275818 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:31.275753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:31.276274 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:31.275831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:31.276748 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:31.276723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e13f0853-dab1-403c-972f-0079739bd48a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:31.276954 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:31.276937    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:31.277093 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:31.277081    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls podName:e13f0853-dab1-403c-972f-0079739bd48a nodeName:}" failed. No retries permitted until 2026-09-18 19:16:47.277047466 +0000 UTC m=+150.995704760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rbnz8" (UID: "e13f0853-dab1-403c-972f-0079739bd48a") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:37.216183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.216155    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"]
Sep 18 19:16:37.219403 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.219387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.223622 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.223598    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 19:16:37.223762 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.223626    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 19:16:37.224422 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.224405    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mq2n4\""
Sep 18 19:16:37.256942 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.256921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"]
Sep 18 19:16:37.272789 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.272769    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588fcb9b4c-kgqck"]
Sep 18 19:16:37.275562 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.275540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.281316 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.281297    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:16:37.281460 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.281434    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:16:37.281579 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.281561    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:16:37.281676 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.281660    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qbqcn\""
Sep 18 19:16:37.290125 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.290103    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:16:37.302506 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.302487    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588fcb9b4c-kgqck"]
Sep 18 19:16:37.322530 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.322507    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/10a41341-2ad5-454e-8d6b-f954f8573c7a-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.322615 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.322542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10a41341-2ad5-454e-8d6b-f954f8573c7a-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.423823 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xx9s\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-kube-api-access-8xx9s\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.423988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-trusted-ca\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.423988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-certificates\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.423988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-bound-sa-token\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.423988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-tls\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.423988 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.423985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-image-registry-private-configuration\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.424224 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.424033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e1f8ab7-0985-4029-b609-acefa3c3fc35-ca-trust-extracted\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.424224 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.424072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/10a41341-2ad5-454e-8d6b-f954f8573c7a-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.424224 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.424102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10a41341-2ad5-454e-8d6b-f954f8573c7a-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.424224 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.424123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-installation-pull-secrets\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.424662 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.424620    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/10a41341-2ad5-454e-8d6b-f954f8573c7a-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.426581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.426559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10a41341-2ad5-454e-8d6b-f954f8573c7a-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-kx2c8\" (UID: \"10a41341-2ad5-454e-8d6b-f954f8573c7a\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.524457 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-image-registry-private-configuration\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524676 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e1f8ab7-0985-4029-b609-acefa3c3fc35-ca-trust-extracted\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524676 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-installation-pull-secrets\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524676 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524567    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx9s\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-kube-api-access-8xx9s\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524676 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-trusted-ca\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524886 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524677    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-certificates\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524886 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-bound-sa-token\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.524886 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524757    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-tls\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.525027 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.524895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e1f8ab7-0985-4029-b609-acefa3c3fc35-ca-trust-extracted\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.525544 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.525522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-certificates\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.525902 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.525878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e1f8ab7-0985-4029-b609-acefa3c3fc35-trusted-ca\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.527269 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.527248    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-registry-tls\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.527354 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.527327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"
Sep 18 19:16:37.527419 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.527351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-installation-pull-secrets\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.527419 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.527353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e1f8ab7-0985-4029-b609-acefa3c3fc35-image-registry-private-configuration\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.545770 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.545740    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-bound-sa-token\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.545946 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.545932    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx9s\" (UniqueName: \"kubernetes.io/projected/9e1f8ab7-0985-4029-b609-acefa3c3fc35-kube-api-access-8xx9s\") pod \"image-registry-588fcb9b4c-kgqck\" (UID: \"9e1f8ab7-0985-4029-b609-acefa3c3fc35\") " pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.584244 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.584140    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:37.814009 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.813934    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8"]
Sep 18 19:16:37.817308 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:37.817286    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a41341_2ad5_454e_8d6b_f954f8573c7a.slice/crio-fd8d3ad914daca6fdfbfe6e9da889554594e2d2fbb1edad8b4e7a0904e30bd08 WatchSource:0}: Error finding container fd8d3ad914daca6fdfbfe6e9da889554594e2d2fbb1edad8b4e7a0904e30bd08: Status 404 returned error can't find the container with id fd8d3ad914daca6fdfbfe6e9da889554594e2d2fbb1edad8b4e7a0904e30bd08
Sep 18 19:16:37.905756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:37.905727    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588fcb9b4c-kgqck"]
Sep 18 19:16:37.909071 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:37.909045    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1f8ab7_0985_4029_b609_acefa3c3fc35.slice/crio-3cb727f09f461724e257c4a33f35296727cb3861806e08b0800d20a0eac75cb0 WatchSource:0}: Error finding container 3cb727f09f461724e257c4a33f35296727cb3861806e08b0800d20a0eac75cb0: Status 404 returned error can't find the container with id 3cb727f09f461724e257c4a33f35296727cb3861806e08b0800d20a0eac75cb0
Sep 18 19:16:38.005456 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:38.005419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck" event={"ID":"9e1f8ab7-0985-4029-b609-acefa3c3fc35","Type":"ContainerStarted","Data":"980f2d32da105b2a5c16dbb32cff5dd79ede0be6a18a1a35f0d3f0b4ea72faa2"}
Sep 18 19:16:38.005572 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:38.005477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck" event={"ID":"9e1f8ab7-0985-4029-b609-acefa3c3fc35","Type":"ContainerStarted","Data":"3cb727f09f461724e257c4a33f35296727cb3861806e08b0800d20a0eac75cb0"}
Sep 18 19:16:38.005572 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:38.005501    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:16:38.006625 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:38.006604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8" event={"ID":"10a41341-2ad5-454e-8d6b-f954f8573c7a","Type":"ContainerStarted","Data":"fd8d3ad914daca6fdfbfe6e9da889554594e2d2fbb1edad8b4e7a0904e30bd08"}
Sep 18 19:16:38.056036 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:38.055988    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck" podStartSLOduration=1.055968432 podStartE2EDuration="1.055968432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:38.049467133 +0000 UTC m=+141.768124434" watchObservedRunningTime="2026-09-18 19:16:38.055968432 +0000 UTC m=+141.774625734"
Sep 18 19:16:40.014005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:40.013966    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8" event={"ID":"10a41341-2ad5-454e-8d6b-f954f8573c7a","Type":"ContainerStarted","Data":"a931cc20ac67428ada849f32b21423cf8c5cc436cdd296442b726a4b455fa11f"}
Sep 18 19:16:40.045401 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:40.042417    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-kx2c8" podStartSLOduration=1.863073617 podStartE2EDuration="3.042398516s" totalImagesPullingTime="1.179324899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:40.038169661 +0000 UTC m=+143.756826963" watchObservedRunningTime="2026-09-18 19:16:40.042398516 +0000 UTC m=+143.761055818"
Sep 18 19:16:41.255468 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:41.255409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:41.258073 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:41.258049    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/58a99a04-199f-409c-80c5-ecf84c2ba538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vrnxw\" (UID: \"58a99a04-199f-409c-80c5-ecf84c2ba538\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:41.283168 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:41.283144    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9c9h4\""
Sep 18 19:16:41.291898 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:41.291884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"
Sep 18 19:16:41.571967 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:41.571885    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw"]
Sep 18 19:16:41.577065 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:41.577036    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a99a04_199f_409c_80c5_ecf84c2ba538.slice/crio-6c897ecc241772980a3759e1a4e286a877b3aa1eaa6a6568cb000a2dcc4de8f5 WatchSource:0}: Error finding container 6c897ecc241772980a3759e1a4e286a877b3aa1eaa6a6568cb000a2dcc4de8f5: Status 404 returned error can't find the container with id 6c897ecc241772980a3759e1a4e286a877b3aa1eaa6a6568cb000a2dcc4de8f5
Sep 18 19:16:42.022484 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:42.022450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw" event={"ID":"58a99a04-199f-409c-80c5-ecf84c2ba538","Type":"ContainerStarted","Data":"6c897ecc241772980a3759e1a4e286a877b3aa1eaa6a6568cb000a2dcc4de8f5"}
Sep 18 19:16:44.030452 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:44.030419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw" event={"ID":"58a99a04-199f-409c-80c5-ecf84c2ba538","Type":"ContainerStarted","Data":"9f940d747d07c0c06a9e3baf4144ff9d1cc86c7cc40ba493f9c74eb40d42bebc"}
Sep 18 19:16:44.052273 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:44.052218    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vrnxw" podStartSLOduration=33.5505463 podStartE2EDuration="35.052199813s" totalImagesPullingTime="1.501653513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:44.051528372 +0000 UTC m=+147.770185672" watchObservedRunningTime="2026-09-18 19:16:44.052199813 +0000 UTC m=+147.770857114"
Sep 18 19:16:47.305769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:47.305719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:47.308163 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:47.308142    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e13f0853-dab1-403c-972f-0079739bd48a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbnz8\" (UID: \"e13f0853-dab1-403c-972f-0079739bd48a\") " pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:47.599925 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:47.599853    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-npv6k\""
Sep 18 19:16:47.608475 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:47.608449    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rbnz8"
Sep 18 19:16:47.958019 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:47.957992    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rbnz8"]
Sep 18 19:16:47.960502 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:47.960475    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f0853_dab1_403c_972f_0079739bd48a.slice/crio-2ef29375d155cde2b184acad19a31d0ba94b20a4c324ddd39d9338a049d23d69 WatchSource:0}: Error finding container 2ef29375d155cde2b184acad19a31d0ba94b20a4c324ddd39d9338a049d23d69: Status 404 returned error can't find the container with id 2ef29375d155cde2b184acad19a31d0ba94b20a4c324ddd39d9338a049d23d69
Sep 18 19:16:48.045395 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:48.045368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbnz8" event={"ID":"e13f0853-dab1-403c-972f-0079739bd48a","Type":"ContainerStarted","Data":"2ef29375d155cde2b184acad19a31d0ba94b20a4c324ddd39d9338a049d23d69"}
Sep 18 19:16:49.049945 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:49.049919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbnz8" event={"ID":"e13f0853-dab1-403c-972f-0079739bd48a","Type":"ContainerStarted","Data":"3693ec8e26414e901e6f4167bf03dc3205b8fa32d5a599b40e99ba295acab009"}
Sep 18 19:16:50.055515 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:50.055478    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbnz8" event={"ID":"e13f0853-dab1-403c-972f-0079739bd48a","Type":"ContainerStarted","Data":"4b482d0786262727c03d7d3d23516e5e6e8b97c37b6d836f0c79243974101819"}
Sep 18 19:16:51.060567 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:51.060530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbnz8" event={"ID":"e13f0853-dab1-403c-972f-0079739bd48a","Type":"ContainerStarted","Data":"9b88839ae226f817242d7f3220e3306df4d966cb302e1416e8b15c097633df15"}
Sep 18 19:16:51.079677 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:51.079612    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rbnz8" podStartSLOduration=33.993604096 podStartE2EDuration="36.079597858s" totalImagesPullingTime="2.085993762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:51.078790028 +0000 UTC m=+154.797447329" watchObservedRunningTime="2026-09-18 19:16:51.079597858 +0000 UTC m=+154.798255158"
Sep 18 19:16:55.278957 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:55.278914    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-flnrv" podUID="64df114d-0390-4d42-b107-656830b8a27c"
Sep 18 19:16:55.538573 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:16:55.538487    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l9829" podUID="738fc24a-cf2d-4ccc-995e-eb9e54e6b044"
Sep 18 19:16:55.587111 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.587083    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bhgv7"]
Sep 18 19:16:55.589256 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.589241    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.596781 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.596761    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:16:55.597465 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.597446    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:16:55.597703 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.597687    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:16:55.597757 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.597703    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:16:55.600716 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.600703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nzvgt\""
Sep 18 19:16:55.768569 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768534    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-tls\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768569 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768792 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768792 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-textfile\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768792 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768728    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnh6f\" (UniqueName: \"kubernetes.io/projected/d22ad3ab-1106-4f5c-9142-83bc73712779-kube-api-access-pnh6f\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768898 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768791    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-wtmp\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768898 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-sys\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768898 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-root\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.768997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.768902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-metrics-client-ca\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869751 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-root\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869751 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-metrics-client-ca\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869751 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-tls\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869799    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-root\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-textfile\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.869972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.869836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnh6f\" (UniqueName: \"kubernetes.io/projected/d22ad3ab-1106-4f5c-9142-83bc73712779-kube-api-access-pnh6f\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870193 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870010    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-wtmp\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870193 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870179    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-wtmp\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870292 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-textfile\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870292 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-sys\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870384 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d22ad3ab-1106-4f5c-9142-83bc73712779-sys\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870384 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-metrics-client-ca\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.870450 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.870415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.872281 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.872265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.872468 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.872451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d22ad3ab-1106-4f5c-9142-83bc73712779-node-exporter-tls\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.885730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.885707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnh6f\" (UniqueName: \"kubernetes.io/projected/d22ad3ab-1106-4f5c-9142-83bc73712779-kube-api-access-pnh6f\") pod \"node-exporter-bhgv7\" (UID: \"d22ad3ab-1106-4f5c-9142-83bc73712779\") " pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.898663 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:55.898612    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhgv7"
Sep 18 19:16:55.908089 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:16:55.908058    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ad3ab_1106_4f5c_9142_83bc73712779.slice/crio-fb2c394595e453c058ca885a2421e1ce4a44c789cdc4ac2e7fcb80c58d3ecd9c WatchSource:0}: Error finding container fb2c394595e453c058ca885a2421e1ce4a44c789cdc4ac2e7fcb80c58d3ecd9c: Status 404 returned error can't find the container with id fb2c394595e453c058ca885a2421e1ce4a44c789cdc4ac2e7fcb80c58d3ecd9c
Sep 18 19:16:56.078537 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:56.078506    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flnrv"
Sep 18 19:16:56.078537 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:56.078532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhgv7" event={"ID":"d22ad3ab-1106-4f5c-9142-83bc73712779","Type":"ContainerStarted","Data":"fb2c394595e453c058ca885a2421e1ce4a44c789cdc4ac2e7fcb80c58d3ecd9c"}
Sep 18 19:16:57.083901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:57.083803    2584 generic.go:356] "Generic (PLEG): container finished" podID="d22ad3ab-1106-4f5c-9142-83bc73712779" containerID="d85f3662608b26000e7bc0cc61e9104ea1c9ef7c193f94923b46bc56b8b2a673" exitCode=0
Sep 18 19:16:57.083901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:57.083862    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhgv7" event={"ID":"d22ad3ab-1106-4f5c-9142-83bc73712779","Type":"ContainerDied","Data":"d85f3662608b26000e7bc0cc61e9104ea1c9ef7c193f94923b46bc56b8b2a673"}
Sep 18 19:16:57.589201 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:57.589170    2584 patch_prober.go:28] interesting pod/image-registry-588fcb9b4c-kgqck container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 19:16:57.589360 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:57.589219    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck" podUID="9e1f8ab7-0985-4029-b609-acefa3c3fc35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:16:58.089608 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:58.089571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhgv7" event={"ID":"d22ad3ab-1106-4f5c-9142-83bc73712779","Type":"ContainerStarted","Data":"67a5b93c8800fce56d801a0eabaa64a00806cc82ded2d829b6dd0b0a541034dc"}
Sep 18 19:16:58.090083 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:58.089625    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhgv7" event={"ID":"d22ad3ab-1106-4f5c-9142-83bc73712779","Type":"ContainerStarted","Data":"a3c6233b8f7c34a670d0851f37ae896875184fc7889b68e8a4b2b0d1893e765d"}
Sep 18 19:16:58.116844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:58.116795    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bhgv7" podStartSLOduration=2.176158431 podStartE2EDuration="3.116779201s" totalImagesPullingTime="872.400782ms" totalInitContainerRuntime="68.219988ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:58.116486068 +0000 UTC m=+161.835143368" watchObservedRunningTime="2026-09-18 19:16:58.116779201 +0000 UTC m=+161.835436502"
Sep 18 19:16:59.013782 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:16:59.013752    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-588fcb9b4c-kgqck"
Sep 18 19:17:00.208094 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:00.208055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:17:00.210906 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:00.210876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/64df114d-0390-4d42-b107-656830b8a27c-metrics-tls\") pod \"dns-default-flnrv\" (UID: \"64df114d-0390-4d42-b107-656830b8a27c\") " pod="openshift-dns/dns-default-flnrv"
Sep 18 19:17:00.283837 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:00.283810    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7n8q\""
Sep 18 19:17:00.289427 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:00.289409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flnrv"
Sep 18 19:17:00.605871 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:00.605841    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flnrv"]
Sep 18 19:17:00.609173 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:17:00.609134    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64df114d_0390_4d42_b107_656830b8a27c.slice/crio-0bc6e4b9e63a83461cb256497c4aa2f786023ced204c35ec75906d789fd54b3f WatchSource:0}: Error finding container 0bc6e4b9e63a83461cb256497c4aa2f786023ced204c35ec75906d789fd54b3f: Status 404 returned error can't find the container with id 0bc6e4b9e63a83461cb256497c4aa2f786023ced204c35ec75906d789fd54b3f
Sep 18 19:17:01.101686 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:01.101650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flnrv" event={"ID":"64df114d-0390-4d42-b107-656830b8a27c","Type":"ContainerStarted","Data":"0bc6e4b9e63a83461cb256497c4aa2f786023ced204c35ec75906d789fd54b3f"}
Sep 18 19:17:03.110229 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.110190    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flnrv" event={"ID":"64df114d-0390-4d42-b107-656830b8a27c","Type":"ContainerStarted","Data":"4f21a257f2fd2092a8bbb7b0b5e907c98f8de178a1c4548171e540d7c8d840ee"}
Sep 18 19:17:03.110610 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.110237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flnrv" event={"ID":"64df114d-0390-4d42-b107-656830b8a27c","Type":"ContainerStarted","Data":"b1535d261d035a64614e6286bfecf344517154d8125014c810c7e1ead580fd80"}
Sep 18 19:17:03.110610 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.110267    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-flnrv"
Sep 18 19:17:03.150824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.150774    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-flnrv" podStartSLOduration=129.644249447 podStartE2EDuration="2m11.150759845s" totalImagesPullingTime="1.506510398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:03.14940475 +0000 UTC m=+166.868062051" watchObservedRunningTime="2026-09-18 19:17:03.150759845 +0000 UTC m=+166.869417142"
Sep 18 19:17:03.241810 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.241781    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:17:03.244859 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.244843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.251077 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.250970    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:17:03.251677 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.251651    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:17:03.251791 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.251774    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:17:03.251933 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.251915    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:17:03.252085 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.252066    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:17:03.252715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.252695    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:17:03.252817 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.252717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:17:03.252817 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.252792    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjztf\""
Sep 18 19:17:03.259577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.259556    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:17:03.269432 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.269409    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:17:03.330909 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.330877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspq9\" (UniqueName: \"kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331052 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.330925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331052 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.330971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331052 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.330992    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331052 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.331045    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331186 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.331072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.331186 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.331086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432115 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432115 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432115 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspq9\" (UniqueName: \"kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.432309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.433046 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432987    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.433046 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.432996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.433240 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.433082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.433322 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.433300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.434890 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.434868    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.434979 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.434912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.446375 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.446356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspq9\" (UniqueName: \"kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9\") pod \"console-56d885ccc-zl9rd\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") " pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.553856 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.553819    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:03.850708 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:03.850675    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:17:03.856294 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:17:03.855911    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b676d8_e212_4226_8821_8fa72d56cd04.slice/crio-577ec35bd6cf1a708cd80ca862a19e40717ddffcd3adceae5baa1bad1fe3e2a1 WatchSource:0}: Error finding container 577ec35bd6cf1a708cd80ca862a19e40717ddffcd3adceae5baa1bad1fe3e2a1: Status 404 returned error can't find the container with id 577ec35bd6cf1a708cd80ca862a19e40717ddffcd3adceae5baa1bad1fe3e2a1
Sep 18 19:17:04.115210 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:04.115113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d885ccc-zl9rd" event={"ID":"59b676d8-e212-4226-8821-8fa72d56cd04","Type":"ContainerStarted","Data":"577ec35bd6cf1a708cd80ca862a19e40717ddffcd3adceae5baa1bad1fe3e2a1"}
Sep 18 19:17:06.505509 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:06.505465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:17:08.133275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:08.133236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d885ccc-zl9rd" event={"ID":"59b676d8-e212-4226-8821-8fa72d56cd04","Type":"ContainerStarted","Data":"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"}
Sep 18 19:17:08.163229 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:08.163187    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d885ccc-zl9rd" podStartSLOduration=1.414282844 podStartE2EDuration="5.163170352s" totalImagesPullingTime="3.748887508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:08.162738089 +0000 UTC m=+171.881395390" watchObservedRunningTime="2026-09-18 19:17:08.163170352 +0000 UTC m=+171.881827652"
Sep 18 19:17:13.118219 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:13.118187    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-flnrv"
Sep 18 19:17:13.554224 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:13.554198    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:13.554367 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:13.554243    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:13.558854 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:13.558827    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:14.158275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:14.158242    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:17:25.194543 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:25.194452    2584 generic.go:356] "Generic (PLEG): container finished" podID="2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8" containerID="d8199707ee9e7acb320f9f33d4d670f75fc53b03ad63579f24e448371f79acb7" exitCode=0
Sep 18 19:17:25.194950 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:25.194533    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd" event={"ID":"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8","Type":"ContainerDied","Data":"d8199707ee9e7acb320f9f33d4d670f75fc53b03ad63579f24e448371f79acb7"}
Sep 18 19:17:25.194950 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:25.194913    2584 scope.go:117] "RemoveContainer" containerID="d8199707ee9e7acb320f9f33d4d670f75fc53b03ad63579f24e448371f79acb7"
Sep 18 19:17:26.076569 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:26.076536    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pfs8t_09842818-d867-4448-b34b-cdded954cb09/serve-healthcheck-canary/0.log"
Sep 18 19:17:26.200155 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:26.200124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-chdrd" event={"ID":"2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8","Type":"ContainerStarted","Data":"dc702dd12bbb6b1230cf7b65a36280e2ac8ec59bc5e708caf9d203d6324ff2a9"}
Sep 18 19:17:40.129438 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:40.129396    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" podUID="fe0a4726-8924-45c7-aea8-bb203ee53c6f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:17:50.128939 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:17:50.128892    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" podUID="fe0a4726-8924-45c7-aea8-bb203ee53c6f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:18:00.129179 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.129128    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" podUID="fe0a4726-8924-45c7-aea8-bb203ee53c6f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:18:00.129774 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.129230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2"
Sep 18 19:18:00.129962 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.129923    2584 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b680e935f9bf2da3b13208dea87de3f2aa4a232f0bf54d9827fb9253682d4c6c"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 18 19:18:00.130047 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.129993    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" podUID="fe0a4726-8924-45c7-aea8-bb203ee53c6f" containerName="service-proxy" containerID="cri-o://b680e935f9bf2da3b13208dea87de3f2aa4a232f0bf54d9827fb9253682d4c6c" gracePeriod=30
Sep 18 19:18:00.326883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.326857    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe0a4726-8924-45c7-aea8-bb203ee53c6f" containerID="b680e935f9bf2da3b13208dea87de3f2aa4a232f0bf54d9827fb9253682d4c6c" exitCode=2
Sep 18 19:18:00.326997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.326916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerDied","Data":"b680e935f9bf2da3b13208dea87de3f2aa4a232f0bf54d9827fb9253682d4c6c"}
Sep 18 19:18:00.326997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:00.326947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ffcf79c45-2ftk2" event={"ID":"fe0a4726-8924-45c7-aea8-bb203ee53c6f","Type":"ContainerStarted","Data":"7eab63ac7da49369bc27ed2bdacb7f8b2cbbd2ef87101ef7ab7400be58ba62cc"}
Sep 18 19:18:30.284756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:30.284656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:18:30.287455 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:30.287423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738fc24a-cf2d-4ccc-995e-eb9e54e6b044-metrics-certs\") pod \"network-metrics-daemon-l9829\" (UID: \"738fc24a-cf2d-4ccc-995e-eb9e54e6b044\") " pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:18:30.508343 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:30.508314    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w2qlj\""
Sep 18 19:18:30.516526 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:30.516492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9829"
Sep 18 19:18:30.818453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:30.818323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9829"]
Sep 18 19:18:30.823854 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:18:30.823827    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738fc24a_cf2d_4ccc_995e_eb9e54e6b044.slice/crio-67e0c5b853a84dcd2e033394c80e2948c870f7fce1d55cf2836ab8c2372db699 WatchSource:0}: Error finding container 67e0c5b853a84dcd2e033394c80e2948c870f7fce1d55cf2836ab8c2372db699: Status 404 returned error can't find the container with id 67e0c5b853a84dcd2e033394c80e2948c870f7fce1d55cf2836ab8c2372db699
Sep 18 19:18:31.442415 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:31.442374    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9829" event={"ID":"738fc24a-cf2d-4ccc-995e-eb9e54e6b044","Type":"ContainerStarted","Data":"67e0c5b853a84dcd2e033394c80e2948c870f7fce1d55cf2836ab8c2372db699"}
Sep 18 19:18:33.452125 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:33.452089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9829" event={"ID":"738fc24a-cf2d-4ccc-995e-eb9e54e6b044","Type":"ContainerStarted","Data":"e218b16802b9ee7b371c63dabb0217bbc876931770b70488e18af11dcb7ca55a"}
Sep 18 19:18:33.452511 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:33.452173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9829" event={"ID":"738fc24a-cf2d-4ccc-995e-eb9e54e6b044","Type":"ContainerStarted","Data":"428256541c5a01031789133226ef6af516f3619fe870ac2af89e79fccca07572"}
Sep 18 19:18:33.493408 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:33.493355    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l9829" podStartSLOduration=253.682404457 podStartE2EDuration="4m15.493339233s" totalImagesPullingTime="1.810934776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:33.491375494 +0000 UTC m=+257.210032794" watchObservedRunningTime="2026-09-18 19:18:33.493339233 +0000 UTC m=+257.211996585"
Sep 18 19:18:35.615102 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:18:35.615072    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:19:00.633813 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.633755    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56d885ccc-zl9rd" podUID="59b676d8-e212-4226-8821-8fa72d56cd04" containerName="console" containerID="cri-o://166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc" gracePeriod=15
Sep 18 19:19:00.869393 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.869368    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:19:00.978340 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978278    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978340 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978323    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978350    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978398    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978425    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pspq9\" (UniqueName: \"kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978446    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978504 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978468    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca\") pod \"59b676d8-e212-4226-8821-8fa72d56cd04\" (UID: \"59b676d8-e212-4226-8821-8fa72d56cd04\") "
Sep 18 19:19:00.978912 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978871    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:00.978912 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978849    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:00.979070 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.978926    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config" (OuterVolumeSpecName: "console-config") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:00.979070 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.979009    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca" (OuterVolumeSpecName: "service-ca") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:00.980824 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.980790    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9" (OuterVolumeSpecName: "kube-api-access-pspq9") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "kube-api-access-pspq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:00.980946 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.980925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:00.980995 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:00.980946    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59b676d8-e212-4226-8821-8fa72d56cd04" (UID: "59b676d8-e212-4226-8821-8fa72d56cd04"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:01.079064 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079043    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-serving-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079064 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079062    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59b676d8-e212-4226-8821-8fa72d56cd04-console-oauth-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079169 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079070    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-trusted-ca-bundle\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079169 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079078    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-oauth-serving-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079169 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079086    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pspq9\" (UniqueName: \"kubernetes.io/projected/59b676d8-e212-4226-8821-8fa72d56cd04-kube-api-access-pspq9\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079169 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079095    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-console-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.079169 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.079102    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59b676d8-e212-4226-8821-8fa72d56cd04-service-ca\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:19:01.549518 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.549490    2584 generic.go:356] "Generic (PLEG): container finished" podID="59b676d8-e212-4226-8821-8fa72d56cd04" containerID="166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc" exitCode=0
Sep 18 19:19:01.549735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.549534    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d885ccc-zl9rd"
Sep 18 19:19:01.549735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.549568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d885ccc-zl9rd" event={"ID":"59b676d8-e212-4226-8821-8fa72d56cd04","Type":"ContainerDied","Data":"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"}
Sep 18 19:19:01.549735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.549604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d885ccc-zl9rd" event={"ID":"59b676d8-e212-4226-8821-8fa72d56cd04","Type":"ContainerDied","Data":"577ec35bd6cf1a708cd80ca862a19e40717ddffcd3adceae5baa1bad1fe3e2a1"}
Sep 18 19:19:01.549735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.549623    2584 scope.go:117] "RemoveContainer" containerID="166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"
Sep 18 19:19:01.557760 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.557741    2584 scope.go:117] "RemoveContainer" containerID="166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"
Sep 18 19:19:01.558027 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:19:01.558010    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc\": container with ID starting with 166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc not found: ID does not exist" containerID="166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"
Sep 18 19:19:01.558078 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.558034    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc"} err="failed to get container status \"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc\": rpc error: code = NotFound desc = could not find container \"166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc\": container with ID starting with 166b97a716bc824b4a6bac724402d9c86ebf222f1e96a2326845cca1f57b05dc not found: ID does not exist"
Sep 18 19:19:01.570361 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.570341    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:19:01.572176 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:01.572158    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56d885ccc-zl9rd"]
Sep 18 19:19:02.509534 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:02.509513    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b676d8-e212-4226-8821-8fa72d56cd04" path="/var/lib/kubelet/pods/59b676d8-e212-4226-8821-8fa72d56cd04/volumes"
Sep 18 19:19:18.361490 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:18.361459    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:19:18.364434 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:18.364411    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:19:18.367182 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:19:18.367157    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:21:18.987259 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.987225    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-hnpkt"]
Sep 18 19:21:18.987730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.987442    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b676d8-e212-4226-8821-8fa72d56cd04" containerName="console"
Sep 18 19:21:18.987730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.987452    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b676d8-e212-4226-8821-8fa72d56cd04" containerName="console"
Sep 18 19:21:18.987730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.987500    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b676d8-e212-4226-8821-8fa72d56cd04" containerName="console"
Sep 18 19:21:18.990235 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.990219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:18.995313 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.995290    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:21:18.995455 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.995437    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:21:18.996194 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.996097    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:21:18.996194 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.996113    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lns52\""
Sep 18 19:21:18.996312 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:18.996231    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:21:19.010052 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.010030    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-hnpkt"]
Sep 18 19:21:19.077305 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.077282    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-certificates\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.077393 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.077311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5g5l\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-kube-api-access-m5g5l\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.177737 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.177715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-certificates\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.177822 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.177744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5g5l\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-kube-api-access-m5g5l\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.180088 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.180071    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-certificates\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.191520 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.191502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5g5l\" (UniqueName: \"kubernetes.io/projected/8e52eb6a-8dd5-4a67-af11-86e5fd662a4f-kube-api-access-m5g5l\") pod \"keda-admission-6c4d864b86-hnpkt\" (UID: \"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f\") " pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.298872 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.298851    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:19.603214 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:21:19.603178    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e52eb6a_8dd5_4a67_af11_86e5fd662a4f.slice/crio-c2d2ccad4aefd9737007b506e92cb4920c318ef226ea64f6b56cb12a0a7c420b WatchSource:0}: Error finding container c2d2ccad4aefd9737007b506e92cb4920c318ef226ea64f6b56cb12a0a7c420b: Status 404 returned error can't find the container with id c2d2ccad4aefd9737007b506e92cb4920c318ef226ea64f6b56cb12a0a7c420b
Sep 18 19:21:19.604756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.604738    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:21:19.604959 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.604933    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-hnpkt"]
Sep 18 19:21:19.979568 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:19.979506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt" event={"ID":"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f","Type":"ContainerStarted","Data":"c2d2ccad4aefd9737007b506e92cb4920c318ef226ea64f6b56cb12a0a7c420b"}
Sep 18 19:21:22.995997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:22.995964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt" event={"ID":"8e52eb6a-8dd5-4a67-af11-86e5fd662a4f","Type":"ContainerStarted","Data":"b176b06f97adcda6ae3fb169fd12d947b12afd743eb3ae4781581c5db09dcd9e"}
Sep 18 19:21:22.996370 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:22.996106    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:23.032655 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:23.032586    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt" podStartSLOduration=2.209875576 podStartE2EDuration="5.032572574s" totalImagesPullingTime="2.822696998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:23.032402264 +0000 UTC m=+426.751059565" watchObservedRunningTime="2026-09-18 19:21:23.032572574 +0000 UTC m=+426.751229874"
Sep 18 19:21:44.002420 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:44.002391    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-hnpkt"
Sep 18 19:21:58.005353 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.005321    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4f9d9869-fdlk9"]
Sep 18 19:21:58.013094 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.013076    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.015769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.015748    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:21:58.016287 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016272    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:21:58.016369 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016290    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjztf\""
Sep 18 19:21:58.016369 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016274    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:21:58.016488 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016471    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:21:58.016557 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016542    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:21:58.016812 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016798    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:21:58.016874 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.016823    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:21:58.027568 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.027541    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:21:58.028774 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.028756    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4f9d9869-fdlk9"]
Sep 18 19:21:58.139826 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139806    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-console-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.139914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rzr\" (UniqueName: \"kubernetes.io/projected/216ef592-2266-4e7c-a766-88e77e072609-kube-api-access-v9rzr\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.139914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139862    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-oauth-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.139914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139883    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-service-ca\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.140009 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139940    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-oauth-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.140009 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.140009 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.139981    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-trusted-ca-bundle\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241145 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-console-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241219 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rzr\" (UniqueName: \"kubernetes.io/projected/216ef592-2266-4e7c-a766-88e77e072609-kube-api-access-v9rzr\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241260 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-oauth-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241295 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-service-ca\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241295 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-oauth-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241378 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241378 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-trusted-ca-bundle\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241759 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241733    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-console-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241906 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241888    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-service-ca\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.241961 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.241902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-oauth-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.242138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.242120    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216ef592-2266-4e7c-a766-88e77e072609-trusted-ca-bundle\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.244370 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.244347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-serving-cert\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.244459 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.244436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/216ef592-2266-4e7c-a766-88e77e072609-console-oauth-config\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.250446 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.250423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rzr\" (UniqueName: \"kubernetes.io/projected/216ef592-2266-4e7c-a766-88e77e072609-kube-api-access-v9rzr\") pod \"console-5d4f9d9869-fdlk9\" (UID: \"216ef592-2266-4e7c-a766-88e77e072609\") " pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.321739 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.321695    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:21:58.596533 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:58.596471    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4f9d9869-fdlk9"]
Sep 18 19:21:58.599169 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:21:58.599137    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216ef592_2266_4e7c_a766_88e77e072609.slice/crio-efcc7ae3d6ee6b0caaf6dd30acf9a1313bab883696fcc85ac088850ef80743af WatchSource:0}: Error finding container efcc7ae3d6ee6b0caaf6dd30acf9a1313bab883696fcc85ac088850ef80743af: Status 404 returned error can't find the container with id efcc7ae3d6ee6b0caaf6dd30acf9a1313bab883696fcc85ac088850ef80743af
Sep 18 19:21:59.118861 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:59.118831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4f9d9869-fdlk9" event={"ID":"216ef592-2266-4e7c-a766-88e77e072609","Type":"ContainerStarted","Data":"0b7bb6a7ca6bfa9443e216690fb168e6a7b1a1a19ce29ef6b83b2e118511cdf0"}
Sep 18 19:21:59.119189 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:59.118884    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4f9d9869-fdlk9" event={"ID":"216ef592-2266-4e7c-a766-88e77e072609","Type":"ContainerStarted","Data":"efcc7ae3d6ee6b0caaf6dd30acf9a1313bab883696fcc85ac088850ef80743af"}
Sep 18 19:21:59.141806 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:21:59.141760    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d4f9d9869-fdlk9" podStartSLOduration=2.141743151 podStartE2EDuration="2.141743151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:21:59.140707753 +0000 UTC m=+462.859365053" watchObservedRunningTime="2026-09-18 19:21:59.141743151 +0000 UTC m=+462.860400450"
Sep 18 19:22:08.322106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:08.322061    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:22:08.322583 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:08.322223    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:22:08.326999 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:08.326980    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:22:09.162270 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:09.162242    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4f9d9869-fdlk9"
Sep 18 19:22:46.216330 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.216299    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5df28"]
Sep 18 19:22:46.219401 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.219386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.221385 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.221364    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:22:46.221559 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.221498    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fvf6p\""
Sep 18 19:22:46.221985 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.221967    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:22:46.222089 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.222017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:22:46.230757 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.230736    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5df28"]
Sep 18 19:22:46.263960 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.263941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxkd\" (UniqueName: \"kubernetes.io/projected/619592f3-8209-4af1-a1d4-241b95c43628-kube-api-access-ggxkd\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.264058 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.263980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/619592f3-8209-4af1-a1d4-241b95c43628-data\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.364457 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.364434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxkd\" (UniqueName: \"kubernetes.io/projected/619592f3-8209-4af1-a1d4-241b95c43628-kube-api-access-ggxkd\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.364561 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.364473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/619592f3-8209-4af1-a1d4-241b95c43628-data\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.364806 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.364790    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/619592f3-8209-4af1-a1d4-241b95c43628-data\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.387283 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.387257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxkd\" (UniqueName: \"kubernetes.io/projected/619592f3-8209-4af1-a1d4-241b95c43628-kube-api-access-ggxkd\") pod \"seaweedfs-64568bcd49-5df28\" (UID: \"619592f3-8209-4af1-a1d4-241b95c43628\") " pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.528797 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.528778    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:46.795837 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:46.795753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5df28"]
Sep 18 19:22:46.798899 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:22:46.798875    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619592f3_8209_4af1_a1d4_241b95c43628.slice/crio-748a4ac33c7f10d03f2314df585b46d7c59248e1b962f56879c4b397ae73a036 WatchSource:0}: Error finding container 748a4ac33c7f10d03f2314df585b46d7c59248e1b962f56879c4b397ae73a036: Status 404 returned error can't find the container with id 748a4ac33c7f10d03f2314df585b46d7c59248e1b962f56879c4b397ae73a036
Sep 18 19:22:47.288905 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:47.288867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5df28" event={"ID":"619592f3-8209-4af1-a1d4-241b95c43628","Type":"ContainerStarted","Data":"748a4ac33c7f10d03f2314df585b46d7c59248e1b962f56879c4b397ae73a036"}
Sep 18 19:22:50.300998 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:50.300965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5df28" event={"ID":"619592f3-8209-4af1-a1d4-241b95c43628","Type":"ContainerStarted","Data":"c80c6a93e42dbf009fc90eddbe7603ee2320e0df4b4e4a1fb34fe311d11c94ca"}
Sep 18 19:22:50.301395 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:50.301045    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:56.311335 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:56.311303    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5df28"
Sep 18 19:22:56.352017 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:22:56.351967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5df28" podStartSLOduration=7.188682892 podStartE2EDuration="10.35194968s" totalImagesPullingTime="3.163266788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:22:50.336684102 +0000 UTC m=+514.055341402" watchObservedRunningTime="2026-09-18 19:22:56.35194968 +0000 UTC m=+520.070607032"
Sep 18 19:24:07.406121 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.406084    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-qvptg"]
Sep 18 19:24:07.408450 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.408429    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:07.411148 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.411125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xhgd9\""
Sep 18 19:24:07.411264 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.411244    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 19:24:07.427501 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.427478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qvptg"]
Sep 18 19:24:07.431223 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.431201    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-5mfdb"]
Sep 18 19:24:07.439415 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.439400    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.443685 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.443665    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:24:07.443770 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.443709    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:24:07.443989 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.443969    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8r9nj\""
Sep 18 19:24:07.447317 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.447299    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5mfdb"]
Sep 18 19:24:07.528368 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.528345    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-metrics-certs\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.528458 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.528371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk9b\" (UniqueName: \"kubernetes.io/projected/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-kube-api-access-fhk9b\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.528458 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.528394    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:07.528458 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.528408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpt4\" (UniqueName: \"kubernetes.io/projected/c0f8df64-fe81-48ab-81a0-5a334fba2994-kube-api-access-jmpt4\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:07.528458 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.528427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.629632 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.629609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-metrics-certs\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.629735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.629655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk9b\" (UniqueName: \"kubernetes.io/projected/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-kube-api-access-fhk9b\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.629735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.629691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:07.629735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.629712    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpt4\" (UniqueName: \"kubernetes.io/projected/c0f8df64-fe81-48ab-81a0-5a334fba2994-kube-api-access-jmpt4\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:07.629884 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.629734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.629936 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:24:07.629893    2584 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 19:24:07.629990 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:24:07.629947    2584 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:24:07.630044 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:24:07.630004    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs podName:c0f8df64-fe81-48ab-81a0-5a334fba2994 nodeName:}" failed. No retries permitted until 2026-09-18 19:24:08.129982796 +0000 UTC m=+591.848640092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs") pod "model-serving-api-7b954ffc95-qvptg" (UID: "c0f8df64-fe81-48ab-81a0-5a334fba2994") : secret "model-serving-api-tls" not found
Sep 18 19:24:07.630044 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:24:07.630025    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert podName:0dad30e9-45b1-45bb-8f3a-2ffb22a98a13 nodeName:}" failed. No retries permitted until 2026-09-18 19:24:08.130014603 +0000 UTC m=+591.848671883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert") pod "odh-model-controller-7b966d7984-5mfdb" (UID: "0dad30e9-45b1-45bb-8f3a-2ffb22a98a13") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:24:07.632318 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.632298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-metrics-certs\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.652796 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.652775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk9b\" (UniqueName: \"kubernetes.io/projected/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-kube-api-access-fhk9b\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:07.655288 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:07.655269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpt4\" (UniqueName: \"kubernetes.io/projected/c0f8df64-fe81-48ab-81a0-5a334fba2994-kube-api-access-jmpt4\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:08.133833 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.133805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:08.133944 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.133837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:08.136002 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.135982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dad30e9-45b1-45bb-8f3a-2ffb22a98a13-cert\") pod \"odh-model-controller-7b966d7984-5mfdb\" (UID: \"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13\") " pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:08.136077 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.136040    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f8df64-fe81-48ab-81a0-5a334fba2994-tls-certs\") pod \"model-serving-api-7b954ffc95-qvptg\" (UID: \"c0f8df64-fe81-48ab-81a0-5a334fba2994\") " pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:08.317955 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.317936    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:08.349141 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.349116    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:08.606034 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.605978    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qvptg"]
Sep 18 19:24:08.608107 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:24:08.608084    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f8df64_fe81_48ab_81a0_5a334fba2994.slice/crio-8fd67aecc389d3b1bb704aac296cfab7c4c53797640aede82cf2656a58fe77d5 WatchSource:0}: Error finding container 8fd67aecc389d3b1bb704aac296cfab7c4c53797640aede82cf2656a58fe77d5: Status 404 returned error can't find the container with id 8fd67aecc389d3b1bb704aac296cfab7c4c53797640aede82cf2656a58fe77d5
Sep 18 19:24:08.651661 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:08.651586    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5mfdb"]
Sep 18 19:24:08.656461 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:24:08.656438    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dad30e9_45b1_45bb_8f3a_2ffb22a98a13.slice/crio-d22ddfa98ab80b38cee6869e95159d3e8a464175f1ba1b66619ce9f776b2bd0d WatchSource:0}: Error finding container d22ddfa98ab80b38cee6869e95159d3e8a464175f1ba1b66619ce9f776b2bd0d: Status 404 returned error can't find the container with id d22ddfa98ab80b38cee6869e95159d3e8a464175f1ba1b66619ce9f776b2bd0d
Sep 18 19:24:09.576608 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:09.576564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5mfdb" event={"ID":"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13","Type":"ContainerStarted","Data":"d22ddfa98ab80b38cee6869e95159d3e8a464175f1ba1b66619ce9f776b2bd0d"}
Sep 18 19:24:09.579416 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:09.579384    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qvptg" event={"ID":"c0f8df64-fe81-48ab-81a0-5a334fba2994","Type":"ContainerStarted","Data":"8fd67aecc389d3b1bb704aac296cfab7c4c53797640aede82cf2656a58fe77d5"}
Sep 18 19:24:12.592807 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.592763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5mfdb" event={"ID":"0dad30e9-45b1-45bb-8f3a-2ffb22a98a13","Type":"ContainerStarted","Data":"59ef8e74368bed4b07051b1a6ef4834186a39b9c0d5c3329c3ba83d2163313bd"}
Sep 18 19:24:12.593242 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.592869    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:12.594185 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.594163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qvptg" event={"ID":"c0f8df64-fe81-48ab-81a0-5a334fba2994","Type":"ContainerStarted","Data":"ac4c12c539fdaf2dde01b34e97ef6b6d379e4df047bcb2397895c080b664a67e"}
Sep 18 19:24:12.594305 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.594232    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:24:12.621769 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.621713    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-5mfdb" podStartSLOduration=2.206448033 podStartE2EDuration="5.621695523s" totalImagesPullingTime="3.41524749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:12.62143094 +0000 UTC m=+596.340088239" watchObservedRunningTime="2026-09-18 19:24:12.621695523 +0000 UTC m=+596.340352815"
Sep 18 19:24:12.652536 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:12.652489    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-qvptg" podStartSLOduration=2.231391968 podStartE2EDuration="5.652475505s" totalImagesPullingTime="3.421083537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:24:12.652008345 +0000 UTC m=+596.370665665" watchObservedRunningTime="2026-09-18 19:24:12.652475505 +0000 UTC m=+596.371132804"
Sep 18 19:24:18.386529 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:18.386502    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:24:18.387333 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:18.387312    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:24:23.600400 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:23.600370    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-5mfdb"
Sep 18 19:24:23.602356 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:24:23.602336    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-qvptg"
Sep 18 19:25:56.840707 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.840672    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:25:56.844354 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.844338    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.846349 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.846326    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6148-predictor-serving-cert\""
Sep 18 19:25:56.846957 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.846940    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:25:56.846957 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.846947    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qfhnz\""
Sep 18 19:25:56.847202 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.846940    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6148-kube-rbac-proxy-sar-config\""
Sep 18 19:25:56.847202 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.847015    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:25:56.853310 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.853289    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:25:56.889458 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.889426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.889572 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.889474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.889572 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.889515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxzj\" (UniqueName: \"kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.990053 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.990020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.990053 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.990056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxzj\" (UniqueName: \"kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.990259 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.990125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.990775 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.990754    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.992582 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.992555    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:56.999384 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:56.999361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxzj\" (UniqueName: \"kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj\") pod \"error-404-isvc-d6148-predictor-66cc88466d-h7tzh\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") " pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:57.133731 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.133652    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:25:57.137483 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.137465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.139496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.139470    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:25:57.139628 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.139470    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:25:57.139628 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.139524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 18 19:25:57.139628 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.139533    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 18 19:25:57.146586 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.146562    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:25:57.155053 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.155035    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:25:57.192142 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.192099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.192326 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.192158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.192326 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.192213    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.192326 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.192292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktptv\" (UniqueName: \"kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.192493 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.192341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.292869 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.292835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.292879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.292910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.292927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktptv\" (UniqueName: \"kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293028 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.292958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:25:57.293062    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 18 19:25:57.293221 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:25:57.293132    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls podName:043d4780-3ddd-417c-85a2-d0db2f33178b nodeName:}" failed. No retries permitted until 2026-09-18 19:25:57.793110472 +0000 UTC m=+701.511767750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls") pod "isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 18 19:25:57.293410 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.293390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293548 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.293530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.293616 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.293601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.303844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.303814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktptv\" (UniqueName: \"kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.427822 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.427797    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:25:57.429926 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:25:57.429904    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902a25e4_b85e_45f7_8be6_c53094fd991b.slice/crio-79c9a9c40a793720d5185a7d1283adb8233abcbd50056b392f9d130a67c4b11e WatchSource:0}: Error finding container 79c9a9c40a793720d5185a7d1283adb8233abcbd50056b392f9d130a67c4b11e: Status 404 returned error can't find the container with id 79c9a9c40a793720d5185a7d1283adb8233abcbd50056b392f9d130a67c4b11e
Sep 18 19:25:57.796401 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.796369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.798944 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.798917    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:57.960285 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:57.960179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerStarted","Data":"79c9a9c40a793720d5185a7d1283adb8233abcbd50056b392f9d130a67c4b11e"}
Sep 18 19:25:58.048913 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:58.048586    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:25:58.376817 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:58.376198    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:25:58.969063 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:25:58.968997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerStarted","Data":"e14a38a0b891f44cfcf31af8e9de3d7354b1e6e826b94575f8a162de9eb7237f"}
Sep 18 19:26:11.029009 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:11.028964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerStarted","Data":"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"}
Sep 18 19:26:11.030737 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:11.030684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerStarted","Data":"6df653ccbb2ee8779ea0723adb471ffcfdf45de9cca2bbc1f4a973363af0f8cd"}
Sep 18 19:26:13.046005 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:13.045966    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerStarted","Data":"d792cd8cc1045347d2739284941b27cbae6edfe4e67ebb9acf54ebd5c5c13146"}
Sep 18 19:26:13.046503 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:13.046186    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:26:13.067251 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:13.067208    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podStartSLOduration=1.718293777 podStartE2EDuration="17.067195366s" totalImagesPullingTime="15.348901589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:13.065024757 +0000 UTC m=+716.783682056" watchObservedRunningTime="2026-09-18 19:26:13.067195366 +0000 UTC m=+716.785852665"
Sep 18 19:26:14.050127 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:14.050100    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:26:14.051428 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:14.051400    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:15.054759 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:15.054728    2584 generic.go:356] "Generic (PLEG): container finished" podID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerID="1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5" exitCode=0
Sep 18 19:26:15.055126 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:15.054804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerDied","Data":"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"}
Sep 18 19:26:15.055196 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:15.055168    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:20.059594 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:20.059560    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:26:20.060162 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:20.060023    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:22.092210 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.092178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerStarted","Data":"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"}
Sep 18 19:26:22.092579 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.092247    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerStarted","Data":"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"}
Sep 18 19:26:22.092653 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.092577    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:26:22.092723 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.092659    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:26:22.093734 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.093706    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:26:22.135786 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:22.135735    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podStartSLOduration=2.739931409 podStartE2EDuration="25.135719451s" totalImagesPullingTime="18.497288863s" totalInitContainerRuntime="3.898499179s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:22.135436329 +0000 UTC m=+725.854093629" watchObservedRunningTime="2026-09-18 19:26:22.135719451 +0000 UTC m=+725.854376751"
Sep 18 19:26:23.096100 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:23.096064    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:26:28.100366 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:28.100333    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:26:28.100914 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:28.100887    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:26:30.060394 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:30.060356    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:38.101454 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:38.101412    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:26:40.061025 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:40.060987    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:48.101017 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:48.100971    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:26:50.060283 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:50.060245    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:26:58.101243 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:26:58.101198    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:27:00.060506 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:00.060479    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:27:08.100935 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:08.100890    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:27:18.100935 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:18.100898    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:27:28.101581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:28.101500    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:27:31.354308 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.354273    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:27:31.354863 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.354556    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container" containerID="cri-o://6df653ccbb2ee8779ea0723adb471ffcfdf45de9cca2bbc1f4a973363af0f8cd" gracePeriod=30
Sep 18 19:27:31.354863 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.354572    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kube-rbac-proxy" containerID="cri-o://d792cd8cc1045347d2739284941b27cbae6edfe4e67ebb9acf54ebd5c5c13146" gracePeriod=30
Sep 18 19:27:31.455603 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.455565    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:27:31.458877 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.458857    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.461738 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:27:31.461711    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"error-404-isvc-0c8e2-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0c8e2-predictor-serving-cert\"" type="*v1.Secret"
Sep 18 19:27:31.461738 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:27:31.461718    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"error-404-isvc-0c8e2-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0c8e2-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 18 19:27:31.461947 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:27:31.461920    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" is forbidden: User \"system:node:ip-10-0-140-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-94.ec2.internal' and this object" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.473461 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.473438    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:27:31.506656 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.506610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4cv\" (UniqueName: \"kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.506767 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.506706    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.506767 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.506753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.607600 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.607522    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.607756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.607602    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.607756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.607675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4cv\" (UniqueName: \"kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:31.616106 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:31.616078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4cv\" (UniqueName: \"kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:32.290927 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.290898    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0c8e2-kube-rbac-proxy-sar-config\""
Sep 18 19:27:32.298341 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.298316    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:32.363699 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.363668    2584 generic.go:356] "Generic (PLEG): container finished" podID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerID="d792cd8cc1045347d2739284941b27cbae6edfe4e67ebb9acf54ebd5c5c13146" exitCode=2
Sep 18 19:27:32.364001 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.363726    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerDied","Data":"d792cd8cc1045347d2739284941b27cbae6edfe4e67ebb9acf54ebd5c5c13146"}
Sep 18 19:27:32.367108 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.367092    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0c8e2-predictor-serving-cert\""
Sep 18 19:27:32.370399 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.370376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls\") pod \"error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") " pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:32.669358 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.669252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:32.940406 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:32.940331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:27:32.943436 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:27:32.943395    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd42eca6_c698_498d_88f3_744c3cde5756.slice/crio-d80d5339008c0fc28d07bbdc409bd2518ae94b66bfec2ba6237bbd1c5574349a WatchSource:0}: Error finding container d80d5339008c0fc28d07bbdc409bd2518ae94b66bfec2ba6237bbd1c5574349a: Status 404 returned error can't find the container with id d80d5339008c0fc28d07bbdc409bd2518ae94b66bfec2ba6237bbd1c5574349a
Sep 18 19:27:33.369133 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.369104    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerStarted","Data":"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"}
Sep 18 19:27:33.369496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.369155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerStarted","Data":"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"}
Sep 18 19:27:33.369496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.369176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerStarted","Data":"d80d5339008c0fc28d07bbdc409bd2518ae94b66bfec2ba6237bbd1c5574349a"}
Sep 18 19:27:33.369496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.369199    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:33.369496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.369218    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:33.370618 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.370596    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:27:33.388764 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:33.388720    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podStartSLOduration=2.388705766 podStartE2EDuration="2.388705766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:33.387111677 +0000 UTC m=+797.105768977" watchObservedRunningTime="2026-09-18 19:27:33.388705766 +0000 UTC m=+797.107363064"
Sep 18 19:27:34.375037 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.375010    2584 generic.go:356] "Generic (PLEG): container finished" podID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerID="6df653ccbb2ee8779ea0723adb471ffcfdf45de9cca2bbc1f4a973363af0f8cd" exitCode=0
Sep 18 19:27:34.375380 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.375085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerDied","Data":"6df653ccbb2ee8779ea0723adb471ffcfdf45de9cca2bbc1f4a973363af0f8cd"}
Sep 18 19:27:34.375533 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.375498    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:27:34.489583 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.489563    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:27:34.531358 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.531335    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config\") pod \"902a25e4-b85e-45f7-8be6-c53094fd991b\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") "
Sep 18 19:27:34.531459 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.531390    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxzj\" (UniqueName: \"kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj\") pod \"902a25e4-b85e-45f7-8be6-c53094fd991b\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") "
Sep 18 19:27:34.531522 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.531497    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls\") pod \"902a25e4-b85e-45f7-8be6-c53094fd991b\" (UID: \"902a25e4-b85e-45f7-8be6-c53094fd991b\") "
Sep 18 19:27:34.531741 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.531715    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "902a25e4-b85e-45f7-8be6-c53094fd991b" (UID: "902a25e4-b85e-45f7-8be6-c53094fd991b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:27:34.533475 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.533454    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj" (OuterVolumeSpecName: "kube-api-access-wpxzj") pod "902a25e4-b85e-45f7-8be6-c53094fd991b" (UID: "902a25e4-b85e-45f7-8be6-c53094fd991b"). InnerVolumeSpecName "kube-api-access-wpxzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:27:34.533559 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.533518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "902a25e4-b85e-45f7-8be6-c53094fd991b" (UID: "902a25e4-b85e-45f7-8be6-c53094fd991b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:34.632931 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.632876    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:27:34.632931 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.632899    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxzj\" (UniqueName: \"kubernetes.io/projected/902a25e4-b85e-45f7-8be6-c53094fd991b-kube-api-access-wpxzj\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:27:34.632931 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:34.632911    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/902a25e4-b85e-45f7-8be6-c53094fd991b-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:27:35.380074 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.380053    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"
Sep 18 19:27:35.380074 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.380055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh" event={"ID":"902a25e4-b85e-45f7-8be6-c53094fd991b","Type":"ContainerDied","Data":"79c9a9c40a793720d5185a7d1283adb8233abcbd50056b392f9d130a67c4b11e"}
Sep 18 19:27:35.380553 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.380119    2584 scope.go:117] "RemoveContainer" containerID="d792cd8cc1045347d2739284941b27cbae6edfe4e67ebb9acf54ebd5c5c13146"
Sep 18 19:27:35.391436 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.391416    2584 scope.go:117] "RemoveContainer" containerID="6df653ccbb2ee8779ea0723adb471ffcfdf45de9cca2bbc1f4a973363af0f8cd"
Sep 18 19:27:35.407292 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.407268    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:27:35.410753 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:35.410732    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6148-predictor-66cc88466d-h7tzh"]
Sep 18 19:27:36.510851 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:36.510824    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" path="/var/lib/kubelet/pods/902a25e4-b85e-45f7-8be6-c53094fd991b/volumes"
Sep 18 19:27:38.101443 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:38.101414    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:27:39.379514 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:39.379475    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:27:39.379996 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:39.379975    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:27:49.380498 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:49.380462    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:27:59.380577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:27:59.380541    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:28:09.380072 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:09.380035    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:28:19.381026 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:19.381004    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:28:21.595196 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.595166    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:28:21.595664 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.595394    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container" containerID="cri-o://522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7" gracePeriod=30
Sep 18 19:28:21.595664 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.595442    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kube-rbac-proxy" containerID="cri-o://77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59" gracePeriod=30
Sep 18 19:28:21.896159 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896091    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:28:21.896385 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896374    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container"
Sep 18 19:28:21.896424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896388    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container"
Sep 18 19:28:21.896424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896398    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kube-rbac-proxy"
Sep 18 19:28:21.896424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896404    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kube-rbac-proxy"
Sep 18 19:28:21.896509 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896462    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kube-rbac-proxy"
Sep 18 19:28:21.896509 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.896471    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="902a25e4-b85e-45f7-8be6-c53094fd991b" containerName="kserve-container"
Sep 18 19:28:21.899286 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.899266    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:21.901399 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.901377    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eff75-predictor-serving-cert\""
Sep 18 19:28:21.901704 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.901686    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eff75-kube-rbac-proxy-sar-config\""
Sep 18 19:28:21.909700 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.909680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:28:21.966545 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.966521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:21.966656 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.966551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:21.966656 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:21.966623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89p8\" (UniqueName: \"kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.067577 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.067552    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.067723 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.067581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.067723 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.067629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89p8\" (UniqueName: \"kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.067837 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:28:22.067787    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-eff75-predictor-serving-cert: secret "error-404-isvc-eff75-predictor-serving-cert" not found
Sep 18 19:28:22.067887 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:28:22.067865    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls podName:0685166e-8e30-4a75-8405-6c5c5265e261 nodeName:}" failed. No retries permitted until 2026-09-18 19:28:22.5678365 +0000 UTC m=+846.286493791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls") pod "error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" (UID: "0685166e-8e30-4a75-8405-6c5c5265e261") : secret "error-404-isvc-eff75-predictor-serving-cert" not found
Sep 18 19:28:22.068155 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.068137    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.077185 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.077164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89p8\" (UniqueName: \"kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.556012 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.555979    2584 generic.go:356] "Generic (PLEG): container finished" podID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerID="77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59" exitCode=2
Sep 18 19:28:22.556164 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.556048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerDied","Data":"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"}
Sep 18 19:28:22.573253 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.573229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.575648 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.575611    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") pod \"error-404-isvc-eff75-predictor-7dcfb97656-vtqk2\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") " pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:22.809547 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:22.809487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:23.082782 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.082708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:28:23.085583 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:28:23.085558    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0685166e_8e30_4a75_8405_6c5c5265e261.slice/crio-f361e4511f0666f45d760d307d12592ba513a26041ac98b698c4807fe37b1e95 WatchSource:0}: Error finding container f361e4511f0666f45d760d307d12592ba513a26041ac98b698c4807fe37b1e95: Status 404 returned error can't find the container with id f361e4511f0666f45d760d307d12592ba513a26041ac98b698c4807fe37b1e95
Sep 18 19:28:23.096849 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.096822    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:28:23.561334 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.561298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerStarted","Data":"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"}
Sep 18 19:28:23.561510 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.561353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerStarted","Data":"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"}
Sep 18 19:28:23.561510 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.561380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerStarted","Data":"f361e4511f0666f45d760d307d12592ba513a26041ac98b698c4807fe37b1e95"}
Sep 18 19:28:23.561620 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.561510    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:23.561620 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.561543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:23.562668 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.562644    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:28:23.582072 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:23.582028    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podStartSLOduration=2.582014074 podStartE2EDuration="2.582014074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:28:23.581952066 +0000 UTC m=+847.300609369" watchObservedRunningTime="2026-09-18 19:28:23.582014074 +0000 UTC m=+847.300671373"
Sep 18 19:28:24.565496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:24.565459    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:28:26.350763 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.350739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:28:26.502878 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.502840    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktptv\" (UniqueName: \"kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv\") pod \"043d4780-3ddd-417c-85a2-d0db2f33178b\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") "
Sep 18 19:28:26.503065 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.502882    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config\") pod \"043d4780-3ddd-417c-85a2-d0db2f33178b\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") "
Sep 18 19:28:26.503065 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.502904    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") pod \"043d4780-3ddd-417c-85a2-d0db2f33178b\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") "
Sep 18 19:28:26.503065 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.502939    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location\") pod \"043d4780-3ddd-417c-85a2-d0db2f33178b\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") "
Sep 18 19:28:26.503065 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.502960    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert\") pod \"043d4780-3ddd-417c-85a2-d0db2f33178b\" (UID: \"043d4780-3ddd-417c-85a2-d0db2f33178b\") "
Sep 18 19:28:26.503307 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.503277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "043d4780-3ddd-417c-85a2-d0db2f33178b" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:28:26.503361 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.503335    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "043d4780-3ddd-417c-85a2-d0db2f33178b" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:28:26.503409 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.503376    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "043d4780-3ddd-417c-85a2-d0db2f33178b" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:28:26.505176 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.505146    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "043d4780-3ddd-417c-85a2-d0db2f33178b" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:28:26.505295 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.505217    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv" (OuterVolumeSpecName: "kube-api-access-ktptv") pod "043d4780-3ddd-417c-85a2-d0db2f33178b" (UID: "043d4780-3ddd-417c-85a2-d0db2f33178b"). InnerVolumeSpecName "kube-api-access-ktptv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:28:26.573918 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.573890    2584 generic.go:356] "Generic (PLEG): container finished" podID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerID="522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7" exitCode=0
Sep 18 19:28:26.574016 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.573927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerDied","Data":"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"}
Sep 18 19:28:26.574016 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.573937    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"
Sep 18 19:28:26.574016 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.573977    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r" event={"ID":"043d4780-3ddd-417c-85a2-d0db2f33178b","Type":"ContainerDied","Data":"e14a38a0b891f44cfcf31af8e9de3d7354b1e6e826b94575f8a162de9eb7237f"}
Sep 18 19:28:26.574016 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.574002    2584 scope.go:117] "RemoveContainer" containerID="77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"
Sep 18 19:28:26.581846 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.581829    2584 scope.go:117] "RemoveContainer" containerID="522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"
Sep 18 19:28:26.589073 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.589056    2584 scope.go:117] "RemoveContainer" containerID="1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"
Sep 18 19:28:26.596110 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.596094    2584 scope.go:117] "RemoveContainer" containerID="77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"
Sep 18 19:28:26.596380 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:28:26.596360    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59\": container with ID starting with 77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59 not found: ID does not exist" containerID="77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"
Sep 18 19:28:26.596473 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.596385    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59"} err="failed to get container status \"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59\": rpc error: code = NotFound desc = could not find container \"77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59\": container with ID starting with 77bc5a0ac462d511f78e1bc0c8fffecd2719f75d016a8569441c90d4aa830f59 not found: ID does not exist"
Sep 18 19:28:26.596473 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.596401    2584 scope.go:117] "RemoveContainer" containerID="522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"
Sep 18 19:28:26.596735 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:28:26.596715    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7\": container with ID starting with 522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7 not found: ID does not exist" containerID="522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"
Sep 18 19:28:26.596820 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.596740    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7"} err="failed to get container status \"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7\": rpc error: code = NotFound desc = could not find container \"522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7\": container with ID starting with 522304954ced01a4973c7ecf1bc47e3e763119e64482eee10f2a09727664b5b7 not found: ID does not exist"
Sep 18 19:28:26.596820 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.596754    2584 scope.go:117] "RemoveContainer" containerID="1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"
Sep 18 19:28:26.597125 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:28:26.597085    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5\": container with ID starting with 1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5 not found: ID does not exist" containerID="1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"
Sep 18 19:28:26.597216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.597134    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5"} err="failed to get container status \"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5\": rpc error: code = NotFound desc = could not find container \"1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5\": container with ID starting with 1549d00b41a64f9874513c9cea4f1d6efec79d3c7ffd1b1e7d804e7a0ca64dd5 not found: ID does not exist"
Sep 18 19:28:26.599007 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.598985    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:28:26.603972 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.603950    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktptv\" (UniqueName: \"kubernetes.io/projected/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-api-access-ktptv\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:28:26.604136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.603975    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:28:26.604136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.603990    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043d4780-3ddd-417c-85a2-d0db2f33178b-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:28:26.604136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.604005    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/043d4780-3ddd-417c-85a2-d0db2f33178b-kserve-provision-location\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:28:26.604136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.604018    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/043d4780-3ddd-417c-85a2-d0db2f33178b-cabundle-cert\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:28:26.607420 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:26.607396    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5fd56cdc95-bcn4r"]
Sep 18 19:28:28.511230 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:28.511196    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" path="/var/lib/kubelet/pods/043d4780-3ddd-417c-85a2-d0db2f33178b/volumes"
Sep 18 19:28:29.570673 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:29.570623    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:28:29.571152 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:29.571098    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:28:39.571617 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:39.571572    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:28:49.571414 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:49.571371    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:28:59.571732 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:28:59.571655    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:29:09.571733 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:29:09.571703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:29:18.413500 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:29:18.413469    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:29:18.414178 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:29:18.414160    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:34:18.440703 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:34:18.440675    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:34:18.443234 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:34:18.442276    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:36:46.786271 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.786185    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:36:46.786779 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.786459    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" containerID="cri-o://0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a" gracePeriod=30
Sep 18 19:36:46.786779 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.786543    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kube-rbac-proxy" containerID="cri-o://1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f" gracePeriod=30
Sep 18 19:36:46.821241 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821214    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:36:46.821547 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821534    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container"
Sep 18 19:36:46.821593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821550    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container"
Sep 18 19:36:46.821593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821565    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kube-rbac-proxy"
Sep 18 19:36:46.821593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821572    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kube-rbac-proxy"
Sep 18 19:36:46.821593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821579    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="storage-initializer"
Sep 18 19:36:46.821593 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821584    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="storage-initializer"
Sep 18 19:36:46.821800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kserve-container"
Sep 18 19:36:46.821800 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.821668    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="043d4780-3ddd-417c-85a2-d0db2f33178b" containerName="kube-rbac-proxy"
Sep 18 19:36:46.824764 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.824747    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.826781 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.826760    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e0eb2-kube-rbac-proxy-sar-config\""
Sep 18 19:36:46.826883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.826781    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e0eb2-predictor-serving-cert\""
Sep 18 19:36:46.836847 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.836826    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:36:46.858845 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.858819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pfv\" (UniqueName: \"kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.858936 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.858865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.858936 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.858917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.960228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.960199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.960395 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.960245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pfv\" (UniqueName: \"kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.960395 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.960282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.960905 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.960881    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.962781 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.962752    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:46.972351 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:46.972328    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pfv\" (UniqueName: \"kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv\") pod \"error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") " pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:47.134256 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:47.134154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:47.411218 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:47.411181    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:36:47.413569 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:36:47.413542    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8d95d7_00fa_4559_aa7e_bb4055a737cd.slice/crio-c78d8d36a8918eb76325229a96f8a3b400399e5b3f4165bdac10e81bdfe56af1 WatchSource:0}: Error finding container c78d8d36a8918eb76325229a96f8a3b400399e5b3f4165bdac10e81bdfe56af1: Status 404 returned error can't find the container with id c78d8d36a8918eb76325229a96f8a3b400399e5b3f4165bdac10e81bdfe56af1
Sep 18 19:36:47.434041 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:47.434018    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd42eca6-c698-498d-88f3-744c3cde5756" containerID="1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f" exitCode=2
Sep 18 19:36:47.434165 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:47.434092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerDied","Data":"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"}
Sep 18 19:36:47.435290 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:47.435269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerStarted","Data":"c78d8d36a8918eb76325229a96f8a3b400399e5b3f4165bdac10e81bdfe56af1"}
Sep 18 19:36:48.441626 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:48.441585    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerStarted","Data":"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"}
Sep 18 19:36:48.442146 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:48.441710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerStarted","Data":"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"}
Sep 18 19:36:48.442146 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:48.441762    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:48.463677 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:48.463612    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podStartSLOduration=2.46359618 podStartE2EDuration="2.46359618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:36:48.461744165 +0000 UTC m=+1352.180401465" watchObservedRunningTime="2026-09-18 19:36:48.46359618 +0000 UTC m=+1352.182253481"
Sep 18 19:36:49.375813 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:49.375767    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 18 19:36:49.380055 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:49.380023    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:36:49.445901 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:49.445871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:49.447218 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:49.447193    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:36:50.428486 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.428461    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:36:50.454768 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.454687    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd42eca6-c698-498d-88f3-744c3cde5756" containerID="0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a" exitCode=0
Sep 18 19:36:50.454768 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.454739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"
Sep 18 19:36:50.455216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.454772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerDied","Data":"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"}
Sep 18 19:36:50.455216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.454825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm" event={"ID":"dd42eca6-c698-498d-88f3-744c3cde5756","Type":"ContainerDied","Data":"d80d5339008c0fc28d07bbdc409bd2518ae94b66bfec2ba6237bbd1c5574349a"}
Sep 18 19:36:50.455216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.454856    2584 scope.go:117] "RemoveContainer" containerID="1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"
Sep 18 19:36:50.455405 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.455383    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:36:50.463843 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.463820    2584 scope.go:117] "RemoveContainer" containerID="0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"
Sep 18 19:36:50.471315 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.471292    2584 scope.go:117] "RemoveContainer" containerID="1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"
Sep 18 19:36:50.471601 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:36:50.471572    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f\": container with ID starting with 1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f not found: ID does not exist" containerID="1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"
Sep 18 19:36:50.471699 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.471607    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f"} err="failed to get container status \"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f\": rpc error: code = NotFound desc = could not find container \"1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f\": container with ID starting with 1f5a1390adc46fa6295360f80b6589d87a568252a3bea7698b3d66131384d37f not found: ID does not exist"
Sep 18 19:36:50.471699 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.471624    2584 scope.go:117] "RemoveContainer" containerID="0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"
Sep 18 19:36:50.471884 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:36:50.471861    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a\": container with ID starting with 0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a not found: ID does not exist" containerID="0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"
Sep 18 19:36:50.471949 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.471890    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a"} err="failed to get container status \"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a\": rpc error: code = NotFound desc = could not find container \"0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a\": container with ID starting with 0466dfe828f33e26f036e81a30d97ab479c5070aeef9ecfdaf41407e6cd3ef1a not found: ID does not exist"
Sep 18 19:36:50.488233 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.488209    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls\") pod \"dd42eca6-c698-498d-88f3-744c3cde5756\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") "
Sep 18 19:36:50.488319 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.488258    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl4cv\" (UniqueName: \"kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv\") pod \"dd42eca6-c698-498d-88f3-744c3cde5756\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") "
Sep 18 19:36:50.488319 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.488295    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config\") pod \"dd42eca6-c698-498d-88f3-744c3cde5756\" (UID: \"dd42eca6-c698-498d-88f3-744c3cde5756\") "
Sep 18 19:36:50.488784 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.488758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd42eca6-c698-498d-88f3-744c3cde5756" (UID: "dd42eca6-c698-498d-88f3-744c3cde5756"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:50.490381 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.490361    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv" (OuterVolumeSpecName: "kube-api-access-xl4cv") pod "dd42eca6-c698-498d-88f3-744c3cde5756" (UID: "dd42eca6-c698-498d-88f3-744c3cde5756"). InnerVolumeSpecName "kube-api-access-xl4cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:50.490493 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.490477    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd42eca6-c698-498d-88f3-744c3cde5756" (UID: "dd42eca6-c698-498d-88f3-744c3cde5756"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:50.589183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.589154    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4cv\" (UniqueName: \"kubernetes.io/projected/dd42eca6-c698-498d-88f3-744c3cde5756-kube-api-access-xl4cv\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:36:50.589183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.589181    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd42eca6-c698-498d-88f3-744c3cde5756-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:36:50.589382 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.589192    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd42eca6-c698-498d-88f3-744c3cde5756-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:36:50.771477 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.771442    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:36:50.774604 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:50.774579    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0c8e2-predictor-ccb4984b6-bmwjm"]
Sep 18 19:36:52.510461 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:52.510429    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" path="/var/lib/kubelet/pods/dd42eca6-c698-498d-88f3-744c3cde5756/volumes"
Sep 18 19:36:55.460230 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:55.460204    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:36:55.460765 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:36:55.460737    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:37:05.460925 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:05.460879    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:37:15.461190 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:15.461148    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:37:25.461589 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:25.461550    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:37:35.461896 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:35.461846    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:37:37.475883 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.475847    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:37:37.476387 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.476085    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" containerID="cri-o://1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf" gracePeriod=30
Sep 18 19:37:37.476387 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.476130    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kube-rbac-proxy" containerID="cri-o://b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874" gracePeriod=30
Sep 18 19:37:37.635235 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.635201    2584 generic.go:356] "Generic (PLEG): container finished" podID="0685166e-8e30-4a75-8405-6c5c5265e261" containerID="b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874" exitCode=2
Sep 18 19:37:37.635398 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.635267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerDied","Data":"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"}
Sep 18 19:37:37.661104 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661076    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:37:37.661394 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661382    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kube-rbac-proxy"
Sep 18 19:37:37.661435 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661397    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kube-rbac-proxy"
Sep 18 19:37:37.661435 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661409    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container"
Sep 18 19:37:37.661435 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661414    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container"
Sep 18 19:37:37.661539 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661461    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kserve-container"
Sep 18 19:37:37.661539 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.661470    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd42eca6-c698-498d-88f3-744c3cde5756" containerName="kube-rbac-proxy"
Sep 18 19:37:37.664451 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.664435    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.666671 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.666631    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6d8ab-predictor-serving-cert\""
Sep 18 19:37:37.667053 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.667035    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6d8ab-kube-rbac-proxy-sar-config\""
Sep 18 19:37:37.677378 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.677356    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:37:37.832807 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.832769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.832982 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.832818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.832982 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.832846    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hfn\" (UniqueName: \"kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.933844 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.933805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.934037 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.933852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.934037 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.933880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hfn\" (UniqueName: \"kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.934506 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.934483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.936532 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.936512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.942993 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.942971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hfn\" (UniqueName: \"kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn\") pod \"error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") " pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:37.974570 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:37.974545    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:38.266970 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.266940    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:37:38.271069 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:37:38.271042    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f0bbdd_6591_4d07_b4fb_626b3a1a1a91.slice/crio-9443d68d2264e5284eac72b687752f934b607b7116d5abbd330c37d7993abf14 WatchSource:0}: Error finding container 9443d68d2264e5284eac72b687752f934b607b7116d5abbd330c37d7993abf14: Status 404 returned error can't find the container with id 9443d68d2264e5284eac72b687752f934b607b7116d5abbd330c37d7993abf14
Sep 18 19:37:38.641576 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.641492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerStarted","Data":"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"}
Sep 18 19:37:38.641576 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.641555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerStarted","Data":"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"}
Sep 18 19:37:38.641576 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.641576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerStarted","Data":"9443d68d2264e5284eac72b687752f934b607b7116d5abbd330c37d7993abf14"}
Sep 18 19:37:38.642000 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.641598    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:38.669218 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:38.669164    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podStartSLOduration=1.6691505100000001 podStartE2EDuration="1.66915051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:38.66789869 +0000 UTC m=+1402.386555991" watchObservedRunningTime="2026-09-18 19:37:38.66915051 +0000 UTC m=+1402.387807811"
Sep 18 19:37:39.566138 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:39.566099    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 19:37:39.571550 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:39.571522    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:37:39.645943 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:39.645906    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:39.647250 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:39.647220    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:37:40.649357 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:40.649316    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:37:40.923305 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:40.923282    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:37:41.060611 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.060572    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config\") pod \"0685166e-8e30-4a75-8405-6c5c5265e261\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") "
Sep 18 19:37:41.060798 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.060725    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x89p8\" (UniqueName: \"kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8\") pod \"0685166e-8e30-4a75-8405-6c5c5265e261\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") "
Sep 18 19:37:41.060798 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.060756    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") pod \"0685166e-8e30-4a75-8405-6c5c5265e261\" (UID: \"0685166e-8e30-4a75-8405-6c5c5265e261\") "
Sep 18 19:37:41.061017 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.060992    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0685166e-8e30-4a75-8405-6c5c5265e261" (UID: "0685166e-8e30-4a75-8405-6c5c5265e261"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:37:41.063010 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.062982    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0685166e-8e30-4a75-8405-6c5c5265e261" (UID: "0685166e-8e30-4a75-8405-6c5c5265e261"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:37:41.063068 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.063011    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8" (OuterVolumeSpecName: "kube-api-access-x89p8") pod "0685166e-8e30-4a75-8405-6c5c5265e261" (UID: "0685166e-8e30-4a75-8405-6c5c5265e261"). InnerVolumeSpecName "kube-api-access-x89p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:37:41.162275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.162180    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89p8\" (UniqueName: \"kubernetes.io/projected/0685166e-8e30-4a75-8405-6c5c5265e261-kube-api-access-x89p8\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:37:41.162275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.162216    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0685166e-8e30-4a75-8405-6c5c5265e261-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:37:41.162275 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.162230    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0685166e-8e30-4a75-8405-6c5c5265e261-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:37:41.654127 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.654089    2584 generic.go:356] "Generic (PLEG): container finished" podID="0685166e-8e30-4a75-8405-6c5c5265e261" containerID="1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf" exitCode=0
Sep 18 19:37:41.654519 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.654144    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"
Sep 18 19:37:41.654519 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.654147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerDied","Data":"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"}
Sep 18 19:37:41.654519 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.654211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2" event={"ID":"0685166e-8e30-4a75-8405-6c5c5265e261","Type":"ContainerDied","Data":"f361e4511f0666f45d760d307d12592ba513a26041ac98b698c4807fe37b1e95"}
Sep 18 19:37:41.654519 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.654236    2584 scope.go:117] "RemoveContainer" containerID="b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"
Sep 18 19:37:41.663175 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.663158    2584 scope.go:117] "RemoveContainer" containerID="1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"
Sep 18 19:37:41.670449 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.670432    2584 scope.go:117] "RemoveContainer" containerID="b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"
Sep 18 19:37:41.670727 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:37:41.670704    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874\": container with ID starting with b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874 not found: ID does not exist" containerID="b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"
Sep 18 19:37:41.670813 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.670733    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874"} err="failed to get container status \"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874\": rpc error: code = NotFound desc = could not find container \"b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874\": container with ID starting with b0e4dca0a2d69bd66cd05f59532a95ce50afa17f58a080b9db6caefd5e2fd874 not found: ID does not exist"
Sep 18 19:37:41.670813 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.670751    2584 scope.go:117] "RemoveContainer" containerID="1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"
Sep 18 19:37:41.671005 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:37:41.670991    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf\": container with ID starting with 1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf not found: ID does not exist" containerID="1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"
Sep 18 19:37:41.671045 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.671009    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf"} err="failed to get container status \"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf\": rpc error: code = NotFound desc = could not find container \"1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf\": container with ID starting with 1081b293c731693e80db9da135319ab3a3dd4898a66e7e80d329e857708e71bf not found: ID does not exist"
Sep 18 19:37:41.676326 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.676303    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:37:41.681241 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:41.681218    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eff75-predictor-7dcfb97656-vtqk2"]
Sep 18 19:37:42.510420 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:42.510378    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" path="/var/lib/kubelet/pods/0685166e-8e30-4a75-8405-6c5c5265e261/volumes"
Sep 18 19:37:45.653715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:45.653685    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:37:45.654173 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:45.654103    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:37:55.654315 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:55.654274    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:37:57.431553 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.431521    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:37:57.431976 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.431768    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container" containerID="cri-o://bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458" gracePeriod=30
Sep 18 19:37:57.431976 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.431853    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kube-rbac-proxy" containerID="cri-o://7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a" gracePeriod=30
Sep 18 19:37:57.618034 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.617969    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:37:57.618269 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618257    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container"
Sep 18 19:37:57.618314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618271    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container"
Sep 18 19:37:57.618314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618285    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kube-rbac-proxy"
Sep 18 19:37:57.618314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618290    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kube-rbac-proxy"
Sep 18 19:37:57.618402 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618345    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kserve-container"
Sep 18 19:37:57.618402 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.618353    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0685166e-8e30-4a75-8405-6c5c5265e261" containerName="kube-rbac-proxy"
Sep 18 19:37:57.621279 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.621263    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.623453 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.623426    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c155b-predictor-serving-cert\""
Sep 18 19:37:57.623601 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.623429    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c155b-kube-rbac-proxy-sar-config\""
Sep 18 19:37:57.629984 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.629960    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:37:57.691243 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.691179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.691243 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.691222    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mzq\" (UniqueName: \"kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.691387 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.691251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.717173 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.717141    2584 generic.go:356] "Generic (PLEG): container finished" podID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerID="7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a" exitCode=2
Sep 18 19:37:57.717308 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.717206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerDied","Data":"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"}
Sep 18 19:37:57.792216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.792186    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mzq\" (UniqueName: \"kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.792342 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.792224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.792342 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.792266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.792433 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:37:57.792348    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c155b-predictor-serving-cert: secret "error-404-isvc-c155b-predictor-serving-cert" not found
Sep 18 19:37:57.792433 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:37:57.792396    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls podName:ebd788b9-793f-4e79-8a99-3029d610ea65 nodeName:}" failed. No retries permitted until 2026-09-18 19:37:58.292379553 +0000 UTC m=+1422.011036833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls") pod "error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" (UID: "ebd788b9-793f-4e79-8a99-3029d610ea65") : secret "error-404-isvc-c155b-predictor-serving-cert" not found
Sep 18 19:37:57.792923 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.792906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:57.801804 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:57.801775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mzq\" (UniqueName: \"kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:58.296386 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:58.296354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:58.298935 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:58.298905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") pod \"error-404-isvc-c155b-predictor-6844b9f65d-8wnx7\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") " pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:58.531965 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:58.531933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:58.804527 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:58.804492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:37:58.808359 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:37:58.808334    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd788b9_793f_4e79_8a99_3029d610ea65.slice/crio-dd9e2569ad4e14b4bab68710700aa4ce99112351fc0af010e3d7db3939b71cd5 WatchSource:0}: Error finding container dd9e2569ad4e14b4bab68710700aa4ce99112351fc0af010e3d7db3939b71cd5: Status 404 returned error can't find the container with id dd9e2569ad4e14b4bab68710700aa4ce99112351fc0af010e3d7db3939b71cd5
Sep 18 19:37:59.726749 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:59.726716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerStarted","Data":"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"}
Sep 18 19:37:59.727150 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:59.726774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerStarted","Data":"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"}
Sep 18 19:37:59.727150 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:59.726795    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerStarted","Data":"dd9e2569ad4e14b4bab68710700aa4ce99112351fc0af010e3d7db3939b71cd5"}
Sep 18 19:37:59.727150 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:59.726834    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:37:59.748610 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:37:59.748550    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podStartSLOduration=2.74853311 podStartE2EDuration="2.74853311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:37:59.747746937 +0000 UTC m=+1423.466404238" watchObservedRunningTime="2026-09-18 19:37:59.74853311 +0000 UTC m=+1423.467190411"
Sep 18 19:38:00.456378 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.456343    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:38:00.678554 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.678532    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:38:00.717946 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.717886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config\") pod \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") "
Sep 18 19:38:00.717946 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.717941    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls\") pod \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") "
Sep 18 19:38:00.718082 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.717965    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2pfv\" (UniqueName: \"kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv\") pod \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\" (UID: \"bc8d95d7-00fa-4559-aa7e-bb4055a737cd\") "
Sep 18 19:38:00.718322 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.718293    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc8d95d7-00fa-4559-aa7e-bb4055a737cd" (UID: "bc8d95d7-00fa-4559-aa7e-bb4055a737cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:38:00.720143 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.720121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv" (OuterVolumeSpecName: "kube-api-access-p2pfv") pod "bc8d95d7-00fa-4559-aa7e-bb4055a737cd" (UID: "bc8d95d7-00fa-4559-aa7e-bb4055a737cd"). InnerVolumeSpecName "kube-api-access-p2pfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:38:00.720238 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.720154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc8d95d7-00fa-4559-aa7e-bb4055a737cd" (UID: "bc8d95d7-00fa-4559-aa7e-bb4055a737cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:38:00.733004 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.732980    2584 generic.go:356] "Generic (PLEG): container finished" podID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerID="bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458" exitCode=0
Sep 18 19:38:00.733347 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.733028    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"
Sep 18 19:38:00.733347 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.733102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerDied","Data":"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"}
Sep 18 19:38:00.733347 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.733154    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh" event={"ID":"bc8d95d7-00fa-4559-aa7e-bb4055a737cd","Type":"ContainerDied","Data":"c78d8d36a8918eb76325229a96f8a3b400399e5b3f4165bdac10e81bdfe56af1"}
Sep 18 19:38:00.733347 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.733186    2584 scope.go:117] "RemoveContainer" containerID="7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"
Sep 18 19:38:00.733541 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.733429    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:38:00.734431 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.734382    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:00.742205 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.742189    2584 scope.go:117] "RemoveContainer" containerID="bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"
Sep 18 19:38:00.749398 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.749384    2584 scope.go:117] "RemoveContainer" containerID="7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"
Sep 18 19:38:00.749660 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:38:00.749621    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a\": container with ID starting with 7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a not found: ID does not exist" containerID="7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"
Sep 18 19:38:00.749770 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.749741    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a"} err="failed to get container status \"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a\": rpc error: code = NotFound desc = could not find container \"7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a\": container with ID starting with 7c01ac92a2509fd2285d96b50ad5d4cca0ccf5ac1d2d6211348e5acc615ade5a not found: ID does not exist"
Sep 18 19:38:00.749816 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.749772    2584 scope.go:117] "RemoveContainer" containerID="bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"
Sep 18 19:38:00.750026 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:38:00.750009    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458\": container with ID starting with bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458 not found: ID does not exist" containerID="bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"
Sep 18 19:38:00.750067 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.750031    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458"} err="failed to get container status \"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458\": rpc error: code = NotFound desc = could not find container \"bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458\": container with ID starting with bfe1bb8960e1fb5a2219d4e361d3c8efeea61c63f2ca44811d7a1969159ad458 not found: ID does not exist"
Sep 18 19:38:00.755546 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.755525    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:38:00.762240 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.762219    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e0eb2-predictor-64f88d65fd-nlbfh"]
Sep 18 19:38:00.818790 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.818766    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:38:00.818790 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.818785    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:38:00.818952 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:00.818796    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pfv\" (UniqueName: \"kubernetes.io/projected/bc8d95d7-00fa-4559-aa7e-bb4055a737cd-kube-api-access-p2pfv\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:38:01.737691 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:01.737651    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:02.510208 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:02.510165    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" path="/var/lib/kubelet/pods/bc8d95d7-00fa-4559-aa7e-bb4055a737cd/volumes"
Sep 18 19:38:05.654151 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:05.654109    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:38:06.742581 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:06.742555    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:38:06.743079 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:06.743050    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:15.654199 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:15.654156    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:38:16.743599 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:16.743561    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:25.654886 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:25.654836    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:38:26.743187 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:26.743144    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:35.655368 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:35.655339    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:38:36.743774 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:36.743739    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:38:46.743781 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:46.743755    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:38:58.603550 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.603519    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:38:58.603965 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.603767    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container" containerID="cri-o://b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437" gracePeriod=30
Sep 18 19:38:58.603965 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.603801    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kube-rbac-proxy" containerID="cri-o://c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7" gracePeriod=30
Sep 18 19:38:58.851965 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.851934    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:38:58.852274 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852261    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container"
Sep 18 19:38:58.852317 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852278    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container"
Sep 18 19:38:58.852317 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852287    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kube-rbac-proxy"
Sep 18 19:38:58.852317 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852294    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kube-rbac-proxy"
Sep 18 19:38:58.852408 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852345    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kserve-container"
Sep 18 19:38:58.852408 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.852352    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8d95d7-00fa-4559-aa7e-bb4055a737cd" containerName="kube-rbac-proxy"
Sep 18 19:38:58.855270 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.855226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:58.858214 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.858191    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad04b-predictor-serving-cert\""
Sep 18 19:38:58.858723 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.858709    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ad04b-kube-rbac-proxy-sar-config\""
Sep 18 19:38:58.869195 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.869175    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:38:58.955969 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.955945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdml\" (UniqueName: \"kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:58.956056 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.955987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:58.956056 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.956030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:58.962944 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.962916    2584 generic.go:356] "Generic (PLEG): container finished" podID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerID="c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7" exitCode=2
Sep 18 19:38:58.963031 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:58.962961    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerDied","Data":"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"}
Sep 18 19:38:59.057146 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.057123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.057233 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.057161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.057233 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.057209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdml\" (UniqueName: \"kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.057922 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.057859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.059620 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.059601    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.067025 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.067004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdml\" (UniqueName: \"kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml\") pod \"error-404-isvc-ad04b-predictor-54cfd54468-pqq2t\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") " pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.165923 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.165865    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:38:59.459359 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.459333    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:38:59.461202 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:38:59.461169    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa60885_1848_4933_bd3a_c6a35498a935.slice/crio-5d6a4e1c5b7e055f01e713a388571253a980ff286c766296437a5baecdc4570f WatchSource:0}: Error finding container 5d6a4e1c5b7e055f01e713a388571253a980ff286c766296437a5baecdc4570f: Status 404 returned error can't find the container with id 5d6a4e1c5b7e055f01e713a388571253a980ff286c766296437a5baecdc4570f
Sep 18 19:38:59.968628 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.968596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerStarted","Data":"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"}
Sep 18 19:38:59.969036 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.968673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerStarted","Data":"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"}
Sep 18 19:38:59.969036 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.968694    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerStarted","Data":"5d6a4e1c5b7e055f01e713a388571253a980ff286c766296437a5baecdc4570f"}
Sep 18 19:38:59.969036 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:38:59.968826    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:39:00.649743 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:00.649684    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 19:39:00.973250 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:00.973175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:39:00.974394 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:00.974367    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:01.758997 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.758975    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:39:01.791346 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.791288    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podStartSLOduration=3.791269147 podStartE2EDuration="3.791269147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:00.003556331 +0000 UTC m=+1483.722213629" watchObservedRunningTime="2026-09-18 19:39:01.791269147 +0000 UTC m=+1485.509926448"
Sep 18 19:39:01.879715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.879627    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8hfn\" (UniqueName: \"kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn\") pod \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") "
Sep 18 19:39:01.879715 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.879700    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls\") pod \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") "
Sep 18 19:39:01.879905 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.879796    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config\") pod \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\" (UID: \"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91\") "
Sep 18 19:39:01.880150 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.880129    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" (UID: "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:01.881778 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.881756    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" (UID: "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:01.881888 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.881872    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn" (OuterVolumeSpecName: "kube-api-access-g8hfn") pod "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" (UID: "d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91"). InnerVolumeSpecName "kube-api-access-g8hfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:01.977953 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.977928    2584 generic.go:356] "Generic (PLEG): container finished" podID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerID="b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437" exitCode=0
Sep 18 19:39:01.978309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.977979    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"
Sep 18 19:39:01.978309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.978020    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerDied","Data":"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"}
Sep 18 19:39:01.978309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.978098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq" event={"ID":"d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91","Type":"ContainerDied","Data":"9443d68d2264e5284eac72b687752f934b607b7116d5abbd330c37d7993abf14"}
Sep 18 19:39:01.978309 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.978139    2584 scope.go:117] "RemoveContainer" containerID="c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"
Sep 18 19:39:01.978568 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.978541    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:01.980256 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.980239    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:39:01.980329 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.980260    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hfn\" (UniqueName: \"kubernetes.io/projected/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-kube-api-access-g8hfn\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:39:01.980329 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.980273    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:39:01.987056 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.987040    2584 scope.go:117] "RemoveContainer" containerID="b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"
Sep 18 19:39:01.994258 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.994239    2584 scope.go:117] "RemoveContainer" containerID="c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"
Sep 18 19:39:01.994515 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:39:01.994499    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7\": container with ID starting with c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7 not found: ID does not exist" containerID="c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"
Sep 18 19:39:01.994553 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.994527    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7"} err="failed to get container status \"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7\": rpc error: code = NotFound desc = could not find container \"c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7\": container with ID starting with c44225ef11ac35c54664372a4f1f95d2387540c0815179b147dec4ac30602af7 not found: ID does not exist"
Sep 18 19:39:01.994553 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.994542    2584 scope.go:117] "RemoveContainer" containerID="b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"
Sep 18 19:39:01.994763 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:39:01.994749    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437\": container with ID starting with b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437 not found: ID does not exist" containerID="b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"
Sep 18 19:39:01.994810 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:01.994766    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437"} err="failed to get container status \"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437\": rpc error: code = NotFound desc = could not find container \"b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437\": container with ID starting with b7f7b1a46358180e2171d1b85589c7e2484b769c1f6e70d19944a77bdb805437 not found: ID does not exist"
Sep 18 19:39:02.001279 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:02.001259    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:39:02.004847 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:02.004828    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6d8ab-predictor-5fdb7d8dd-np9jq"]
Sep 18 19:39:02.512154 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:02.512128    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" path="/var/lib/kubelet/pods/d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91/volumes"
Sep 18 19:39:06.982728 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:06.982697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:39:06.983285 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:06.983256    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:16.983663 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:16.983586    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:18.469091 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:18.468988    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:39:18.486749 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:18.471756    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:39:26.983826 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:26.983784    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:36.983496 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:36.983404    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:39:46.984752 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:39:46.984717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:44:18.493881 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:44:18.493765    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:44:18.499970 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:44:18.498165    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:47:13.414805 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.414769    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415114    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kube-rbac-proxy"
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415126    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kube-rbac-proxy"
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415159    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container"
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415169    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container"
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415216    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kserve-container"
Sep 18 19:47:13.415314 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.415226    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f0bbdd-6591-4d07-b4fb-626b3a1a1a91" containerName="kube-rbac-proxy"
Sep 18 19:47:13.418178 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.418158    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.426265 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.426242    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0596f-predictor-serving-cert\""
Sep 18 19:47:13.426678 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.426631    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0596f-kube-rbac-proxy-sar-config\""
Sep 18 19:47:13.443161 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.443136    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:47:13.528268 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.528240    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:47:13.528475 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.528450    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container" containerID="cri-o://d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df" gracePeriod=30
Sep 18 19:47:13.528590 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.528547    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kube-rbac-proxy" containerID="cri-o://01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83" gracePeriod=30
Sep 18 19:47:13.539521 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:47:13.539498    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd788b9_793f_4e79_8a99_3029d610ea65.slice/crio-01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 19:47:13.592801 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.592771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntbb\" (UniqueName: \"kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.592937 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.592820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.592937 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.592843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.693845 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.693764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntbb\" (UniqueName: \"kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.693845 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.693816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.694058 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.694031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.694507 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.694489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.696503 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.696486    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.702687 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.702629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntbb\" (UniqueName: \"kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb\") pod \"error-404-isvc-0596f-predictor-86cbc5f798-n9lx8\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") " pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.728522 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.728495    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:13.936495 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.936465    2584 generic.go:356] "Generic (PLEG): container finished" podID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerID="01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83" exitCode=2
Sep 18 19:47:13.936659 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:13.936543    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerDied","Data":"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"}
Sep 18 19:47:14.008775 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.008612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:47:14.011345 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:47:14.011320    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40d0190_4b0f_4097_ab43_d459fa5b2c8e.slice/crio-ffc44a6c10871d5c2b65fda6d3effb38ab9b53ad3cbbf7fa51089279a94855a9 WatchSource:0}: Error finding container ffc44a6c10871d5c2b65fda6d3effb38ab9b53ad3cbbf7fa51089279a94855a9: Status 404 returned error can't find the container with id ffc44a6c10871d5c2b65fda6d3effb38ab9b53ad3cbbf7fa51089279a94855a9
Sep 18 19:47:14.942048 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.942010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerStarted","Data":"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"}
Sep 18 19:47:14.942424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.942072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerStarted","Data":"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"}
Sep 18 19:47:14.942424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.942223    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:14.942424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.942293    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:14.942424 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.942316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerStarted","Data":"ffc44a6c10871d5c2b65fda6d3effb38ab9b53ad3cbbf7fa51089279a94855a9"}
Sep 18 19:47:14.943623 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.943594    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:47:14.962104 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:14.962056    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podStartSLOduration=1.962042015 podStartE2EDuration="1.962042015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:14.960432211 +0000 UTC m=+1978.679089512" watchObservedRunningTime="2026-09-18 19:47:14.962042015 +0000 UTC m=+1978.680699315"
Sep 18 19:47:15.946177 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:15.946136    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:47:16.573865 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.573836    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:47:16.721197 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.721124    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config\") pod \"ebd788b9-793f-4e79-8a99-3029d610ea65\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") "
Sep 18 19:47:16.721197 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.721173    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") pod \"ebd788b9-793f-4e79-8a99-3029d610ea65\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") "
Sep 18 19:47:16.721374 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.721295    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7mzq\" (UniqueName: \"kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq\") pod \"ebd788b9-793f-4e79-8a99-3029d610ea65\" (UID: \"ebd788b9-793f-4e79-8a99-3029d610ea65\") "
Sep 18 19:47:16.721522 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.721494    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebd788b9-793f-4e79-8a99-3029d610ea65" (UID: "ebd788b9-793f-4e79-8a99-3029d610ea65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:16.721623 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.721570    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:47:16.723297 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.723270    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebd788b9-793f-4e79-8a99-3029d610ea65" (UID: "ebd788b9-793f-4e79-8a99-3029d610ea65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:16.723385 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.723309    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq" (OuterVolumeSpecName: "kube-api-access-h7mzq") pod "ebd788b9-793f-4e79-8a99-3029d610ea65" (UID: "ebd788b9-793f-4e79-8a99-3029d610ea65"). InnerVolumeSpecName "kube-api-access-h7mzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:16.821924 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.821897    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebd788b9-793f-4e79-8a99-3029d610ea65-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:47:16.821924 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.821921    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mzq\" (UniqueName: \"kubernetes.io/projected/ebd788b9-793f-4e79-8a99-3029d610ea65-kube-api-access-h7mzq\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:47:16.951711 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.951681    2584 generic.go:356] "Generic (PLEG): container finished" podID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerID="d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df" exitCode=0
Sep 18 19:47:16.951994 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.951739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"
Sep 18 19:47:16.951994 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.951755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerDied","Data":"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"}
Sep 18 19:47:16.951994 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.951806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7" event={"ID":"ebd788b9-793f-4e79-8a99-3029d610ea65","Type":"ContainerDied","Data":"dd9e2569ad4e14b4bab68710700aa4ce99112351fc0af010e3d7db3939b71cd5"}
Sep 18 19:47:16.951994 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.951831    2584 scope.go:117] "RemoveContainer" containerID="01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"
Sep 18 19:47:16.960258 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.960240    2584 scope.go:117] "RemoveContainer" containerID="d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"
Sep 18 19:47:16.967336 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.967319    2584 scope.go:117] "RemoveContainer" containerID="01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"
Sep 18 19:47:16.967602 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:47:16.967581    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83\": container with ID starting with 01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83 not found: ID does not exist" containerID="01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"
Sep 18 19:47:16.967682 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.967610    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83"} err="failed to get container status \"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83\": rpc error: code = NotFound desc = could not find container \"01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83\": container with ID starting with 01bb5cc6a2c39109d3af9895b830e3d46381d95cef774ad106593d4e2cbb5f83 not found: ID does not exist"
Sep 18 19:47:16.967682 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.967627    2584 scope.go:117] "RemoveContainer" containerID="d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"
Sep 18 19:47:16.967900 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:47:16.967881    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df\": container with ID starting with d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df not found: ID does not exist" containerID="d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"
Sep 18 19:47:16.967946 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.967906    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df"} err="failed to get container status \"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df\": rpc error: code = NotFound desc = could not find container \"d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df\": container with ID starting with d130103441feef84f543585686f9a3c30033570075784168a8ce45d91ce733df not found: ID does not exist"
Sep 18 19:47:16.976366 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.976319    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:47:16.978675 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:16.978656    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c155b-predictor-6844b9f65d-8wnx7"]
Sep 18 19:47:18.510814 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:18.510782    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" path="/var/lib/kubelet/pods/ebd788b9-793f-4e79-8a99-3029d610ea65/volumes"
Sep 18 19:47:20.951843 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:20.951816    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:47:20.952267 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:20.952224    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:47:30.952874 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:30.952826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:47:40.952868 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:40.952823    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:47:50.952360 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:47:50.952318    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:48:00.952372 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:00.952331    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:48:10.952822 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:10.952791    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:48:13.967807 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:13.967772    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:48:13.968183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:13.967995    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container" containerID="cri-o://9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb" gracePeriod=30
Sep 18 19:48:13.968183 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:13.968032    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kube-rbac-proxy" containerID="cri-o://a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d" gracePeriod=30
Sep 18 19:48:14.175240 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.175210    2584 generic.go:356] "Generic (PLEG): container finished" podID="7aa60885-1848-4933-bd3a-c6a35498a935" containerID="a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d" exitCode=2
Sep 18 19:48:14.175400 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.175290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerDied","Data":"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"}
Sep 18 19:48:14.202817 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.202792    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 19:48:14.203218 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203199    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kube-rbac-proxy"
Sep 18 19:48:14.203259 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203226    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kube-rbac-proxy"
Sep 18 19:48:14.203259 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203249    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container"
Sep 18 19:48:14.203259 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203256    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container"
Sep 18 19:48:14.203353 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203324    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kube-rbac-proxy"
Sep 18 19:48:14.203353 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.203337    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd788b9-793f-4e79-8a99-3029d610ea65" containerName="kserve-container"
Sep 18 19:48:14.206614 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.206597    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.208948 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.208919    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-38355-predictor-serving-cert\""
Sep 18 19:48:14.209063 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.208925    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-38355-kube-rbac-proxy-sar-config\""
Sep 18 19:48:14.220155 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.220107    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 19:48:14.349654 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.349611    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.349755 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.349733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggzr\" (UniqueName: \"kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.349797 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.349762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.450482 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.450456    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggzr\" (UniqueName: \"kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.450592 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.450492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.450592 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.450549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.450745 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:14.450671    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-38355-predictor-serving-cert: secret "error-404-isvc-38355-predictor-serving-cert" not found
Sep 18 19:48:14.450745 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:14.450720    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls podName:9aaa8d21-598f-4f7d-8c21-d974969b3961 nodeName:}" failed. No retries permitted until 2026-09-18 19:48:14.950705521 +0000 UTC m=+2038.669362813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls") pod "error-404-isvc-38355-predictor-64ffd94458-8hgfc" (UID: "9aaa8d21-598f-4f7d-8c21-d974969b3961") : secret "error-404-isvc-38355-predictor-serving-cert" not found
Sep 18 19:48:14.451202 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.451180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.465889 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.465869    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggzr\" (UniqueName: \"kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.954584 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.954534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:14.957353 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:14.957331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") pod \"error-404-isvc-38355-predictor-64ffd94458-8hgfc\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") " pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:15.116313 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:15.116276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:15.393263 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:15.393239    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 19:48:15.395532 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:48:15.395503    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaa8d21_598f_4f7d_8c21_d974969b3961.slice/crio-2466ca25b6d863e90c06ba08cba632ec79e0bfc297f8f4f4f41f0a34512cc32c WatchSource:0}: Error finding container 2466ca25b6d863e90c06ba08cba632ec79e0bfc297f8f4f4f41f0a34512cc32c: Status 404 returned error can't find the container with id 2466ca25b6d863e90c06ba08cba632ec79e0bfc297f8f4f4f41f0a34512cc32c
Sep 18 19:48:16.187136 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.187099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerStarted","Data":"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"}
Sep 18 19:48:16.187561 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.187164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerStarted","Data":"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"}
Sep 18 19:48:16.187561 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.187194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerStarted","Data":"2466ca25b6d863e90c06ba08cba632ec79e0bfc297f8f4f4f41f0a34512cc32c"}
Sep 18 19:48:16.187561 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.187265    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:16.206446 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.206393    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podStartSLOduration=2.206376627 podStartE2EDuration="2.206376627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:16.205416996 +0000 UTC m=+2039.924074297" watchObservedRunningTime="2026-09-18 19:48:16.206376627 +0000 UTC m=+2039.925033939"
Sep 18 19:48:16.920178 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.920152    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:48:16.971228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.971171    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prdml\" (UniqueName: \"kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml\") pod \"7aa60885-1848-4933-bd3a-c6a35498a935\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") "
Sep 18 19:48:16.971228 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.971214    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls\") pod \"7aa60885-1848-4933-bd3a-c6a35498a935\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") "
Sep 18 19:48:16.971400 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.971231    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config\") pod \"7aa60885-1848-4933-bd3a-c6a35498a935\" (UID: \"7aa60885-1848-4933-bd3a-c6a35498a935\") "
Sep 18 19:48:16.971599 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.971576    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7aa60885-1848-4933-bd3a-c6a35498a935" (UID: "7aa60885-1848-4933-bd3a-c6a35498a935"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:16.973432 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.973396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa60885-1848-4933-bd3a-c6a35498a935" (UID: "7aa60885-1848-4933-bd3a-c6a35498a935"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:16.973517 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:16.973437    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml" (OuterVolumeSpecName: "kube-api-access-prdml") pod "7aa60885-1848-4933-bd3a-c6a35498a935" (UID: "7aa60885-1848-4933-bd3a-c6a35498a935"). InnerVolumeSpecName "kube-api-access-prdml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:17.072145 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.072120    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdml\" (UniqueName: \"kubernetes.io/projected/7aa60885-1848-4933-bd3a-c6a35498a935-kube-api-access-prdml\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:17.072145 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.072140    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa60885-1848-4933-bd3a-c6a35498a935-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:17.072145 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.072149    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa60885-1848-4933-bd3a-c6a35498a935-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:17.192614 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.192581    2584 generic.go:356] "Generic (PLEG): container finished" podID="7aa60885-1848-4933-bd3a-c6a35498a935" containerID="9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb" exitCode=0
Sep 18 19:48:17.192963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.192631    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"
Sep 18 19:48:17.192963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.192681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerDied","Data":"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"}
Sep 18 19:48:17.192963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.192744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t" event={"ID":"7aa60885-1848-4933-bd3a-c6a35498a935","Type":"ContainerDied","Data":"5d6a4e1c5b7e055f01e713a388571253a980ff286c766296437a5baecdc4570f"}
Sep 18 19:48:17.192963 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.192773    2584 scope.go:117] "RemoveContainer" containerID="a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"
Sep 18 19:48:17.193167 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.193020    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:17.194587 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.194562    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:17.205650 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.205593    2584 scope.go:117] "RemoveContainer" containerID="9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"
Sep 18 19:48:17.213995 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.213982    2584 scope.go:117] "RemoveContainer" containerID="a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"
Sep 18 19:48:17.214274 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:17.214253    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d\": container with ID starting with a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d not found: ID does not exist" containerID="a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"
Sep 18 19:48:17.214340 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.214281    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d"} err="failed to get container status \"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d\": rpc error: code = NotFound desc = could not find container \"a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d\": container with ID starting with a4165392ecd485b97836a4dc76e822284b5f8dfab30db157071d2a1aac0c7a1d not found: ID does not exist"
Sep 18 19:48:17.214340 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.214296    2584 scope.go:117] "RemoveContainer" containerID="9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"
Sep 18 19:48:17.214562 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:17.214544    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb\": container with ID starting with 9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb not found: ID does not exist" containerID="9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"
Sep 18 19:48:17.214599 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.214569    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb"} err="failed to get container status \"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb\": rpc error: code = NotFound desc = could not find container \"9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb\": container with ID starting with 9ec5fca21b6c8d4c4663dcc6994e49aa1e549d219a284010306d513371eb2ffb not found: ID does not exist"
Sep 18 19:48:17.227735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.227675    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:48:17.228995 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:17.228975    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ad04b-predictor-54cfd54468-pqq2t"]
Sep 18 19:48:18.198428 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:18.198377    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:18.510978 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:18.510947    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" path="/var/lib/kubelet/pods/7aa60885-1848-4933-bd3a-c6a35498a935/volumes"
Sep 18 19:48:23.203831 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:23.203626    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:48:23.204440 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:23.204399    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:28.322432 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.322355    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:48:28.322863 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.322602    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" containerID="cri-o://bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83" gracePeriod=30
Sep 18 19:48:28.322863 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.322681    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kube-rbac-proxy" containerID="cri-o://934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761" gracePeriod=30
Sep 18 19:48:28.699498 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699414    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:48:28.699756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699741    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container"
Sep 18 19:48:28.699756 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699756    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container"
Sep 18 19:48:28.699876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699770    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kube-rbac-proxy"
Sep 18 19:48:28.699876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699775    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kube-rbac-proxy"
Sep 18 19:48:28.699876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699835    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kserve-container"
Sep 18 19:48:28.699876 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.699846    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa60885-1848-4933-bd3a-c6a35498a935" containerName="kube-rbac-proxy"
Sep 18 19:48:28.702820 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.702798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.708685 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.705453    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa781-predictor-serving-cert\""
Sep 18 19:48:28.708685 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.705707    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa781-kube-rbac-proxy-sar-config\""
Sep 18 19:48:28.720408 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.720381    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:48:28.853987 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.853963    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.854113 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.854012    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.854113 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.854043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvtw\" (UniqueName: \"kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.954782 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.954720    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.954782 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.954762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvtw\" (UniqueName: \"kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.954939 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.954805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.954939 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:28.954922    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-aa781-predictor-serving-cert: secret "error-404-isvc-aa781-predictor-serving-cert" not found
Sep 18 19:48:28.955011 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:28.954989    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls podName:81e0676f-066f-4f4f-bfdb-aa9bed853758 nodeName:}" failed. No retries permitted until 2026-09-18 19:48:29.454973476 +0000 UTC m=+2053.173630777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls") pod "error-404-isvc-aa781-predictor-549d86d794-2q82s" (UID: "81e0676f-066f-4f4f-bfdb-aa9bed853758") : secret "error-404-isvc-aa781-predictor-serving-cert" not found
Sep 18 19:48:28.955325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.955306    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:28.969749 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:28.969730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvtw\" (UniqueName: \"kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:29.241897 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.241822    2584 generic.go:356] "Generic (PLEG): container finished" podID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerID="934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761" exitCode=2
Sep 18 19:48:29.242025 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.241897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerDied","Data":"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"}
Sep 18 19:48:29.458613 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.458576    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:29.461313 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.461288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") pod \"error-404-isvc-aa781-predictor-549d86d794-2q82s\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") " pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:29.619819 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.619785    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:29.897110 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:29.897082    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:48:29.899009 ip-10-0-140-94 kubenswrapper[2584]: W0918 19:48:29.898978    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e0676f_066f_4f4f_bfdb_aa9bed853758.slice/crio-e10100e98080fe3eb446d2734eb9bc03f73dcd01af82b69f3fc7859a98f74e7b WatchSource:0}: Error finding container e10100e98080fe3eb446d2734eb9bc03f73dcd01af82b69f3fc7859a98f74e7b: Status 404 returned error can't find the container with id e10100e98080fe3eb446d2734eb9bc03f73dcd01af82b69f3fc7859a98f74e7b
Sep 18 19:48:30.247730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.247653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerStarted","Data":"e8c74277bd601b37bf1df15d52bf272c22be0eb3cda7ae909d46d6aa5c215d4a"}
Sep 18 19:48:30.247730 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.247710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerStarted","Data":"da27373e08cf2f5a13ac163fa194c372208c8ed19302cbe46135a1db9c85200c"}
Sep 18 19:48:30.247889 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.247731    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerStarted","Data":"e10100e98080fe3eb446d2734eb9bc03f73dcd01af82b69f3fc7859a98f74e7b"}
Sep 18 19:48:30.247889 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.247796    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:30.270432 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.270379    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podStartSLOduration=2.270363649 podStartE2EDuration="2.270363649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:30.269958477 +0000 UTC m=+2053.988615779" watchObservedRunningTime="2026-09-18 19:48:30.270363649 +0000 UTC m=+2053.989020953"
Sep 18 19:48:30.946971 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.946931    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 19:48:30.952274 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:30.952246    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:48:31.252198 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.252168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:31.253389 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.253364    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:48:31.572911 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.572890    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:48:31.674422 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.674395    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config\") pod \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") "
Sep 18 19:48:31.674573 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.674452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jntbb\" (UniqueName: \"kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb\") pod \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") "
Sep 18 19:48:31.674573 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.674514    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls\") pod \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\" (UID: \"f40d0190-4b0f-4097-ab43-d459fa5b2c8e\") "
Sep 18 19:48:31.674767 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.674747    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f40d0190-4b0f-4097-ab43-d459fa5b2c8e" (UID: "f40d0190-4b0f-4097-ab43-d459fa5b2c8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:31.676810 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.676784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb" (OuterVolumeSpecName: "kube-api-access-jntbb") pod "f40d0190-4b0f-4097-ab43-d459fa5b2c8e" (UID: "f40d0190-4b0f-4097-ab43-d459fa5b2c8e"). InnerVolumeSpecName "kube-api-access-jntbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:31.676916 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.676812    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f40d0190-4b0f-4097-ab43-d459fa5b2c8e" (UID: "f40d0190-4b0f-4097-ab43-d459fa5b2c8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:31.775160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.775132    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntbb\" (UniqueName: \"kubernetes.io/projected/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-api-access-jntbb\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:31.775160 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.775156    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:31.775299 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:31.775166    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f40d0190-4b0f-4097-ab43-d459fa5b2c8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.257322 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257286    2584 generic.go:356] "Generic (PLEG): container finished" podID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerID="bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83" exitCode=0
Sep 18 19:48:32.257713 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257344    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"
Sep 18 19:48:32.257713 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257373    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerDied","Data":"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"}
Sep 18 19:48:32.257713 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8" event={"ID":"f40d0190-4b0f-4097-ab43-d459fa5b2c8e","Type":"ContainerDied","Data":"ffc44a6c10871d5c2b65fda6d3effb38ab9b53ad3cbbf7fa51089279a94855a9"}
Sep 18 19:48:32.257713 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257463    2584 scope.go:117] "RemoveContainer" containerID="934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"
Sep 18 19:48:32.257894 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.257873    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:48:32.266494 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.266477    2584 scope.go:117] "RemoveContainer" containerID="bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"
Sep 18 19:48:32.273944 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.273918    2584 scope.go:117] "RemoveContainer" containerID="934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"
Sep 18 19:48:32.274264 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:32.274245    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761\": container with ID starting with 934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761 not found: ID does not exist" containerID="934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"
Sep 18 19:48:32.274325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.274271    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761"} err="failed to get container status \"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761\": rpc error: code = NotFound desc = could not find container \"934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761\": container with ID starting with 934c795cb84d49549862a2dcba21ebd7adf47288ccd5be4480a5731ffa601761 not found: ID does not exist"
Sep 18 19:48:32.274325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.274288    2584 scope.go:117] "RemoveContainer" containerID="bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"
Sep 18 19:48:32.274524 ip-10-0-140-94 kubenswrapper[2584]: E0918 19:48:32.274505    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83\": container with ID starting with bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83 not found: ID does not exist" containerID="bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"
Sep 18 19:48:32.274565 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.274532    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83"} err="failed to get container status \"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83\": rpc error: code = NotFound desc = could not find container \"bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83\": container with ID starting with bd67ca070ade8e667d9095399c4c1101d78f9530b94caca25e7ec7ca031d5c83 not found: ID does not exist"
Sep 18 19:48:32.279153 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.279133    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:48:32.283871 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.283852    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0596f-predictor-86cbc5f798-n9lx8"]
Sep 18 19:48:32.510386 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:32.510316    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" path="/var/lib/kubelet/pods/f40d0190-4b0f-4097-ab43-d459fa5b2c8e/volumes"
Sep 18 19:48:33.205012 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:33.204971    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:37.261989 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:37.261959    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:48:37.262526 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:37.262496    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:48:43.204968 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:43.204927    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:47.262509 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:47.262458    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:48:53.204712 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:53.204665    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:48:57.263279 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:48:57.263238    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:49:03.204830 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:49:03.204796    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 19:49:07.262778 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:49:07.262733    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:49:17.263658 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:49:17.263598    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:49:18.523407 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:49:18.523379    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:49:18.528813 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:49:18.528793    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:54:18.547480 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:54:18.547364    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:54:18.552771 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:54:18.552755    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:57:43.633461 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:43.633385    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:57:43.633955 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:43.633618    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container" containerID="cri-o://da27373e08cf2f5a13ac163fa194c372208c8ed19302cbe46135a1db9c85200c" gracePeriod=30
Sep 18 19:57:43.633955 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:43.633704    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kube-rbac-proxy" containerID="cri-o://e8c74277bd601b37bf1df15d52bf272c22be0eb3cda7ae909d46d6aa5c215d4a" gracePeriod=30
Sep 18 19:57:44.437051 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:44.437013    2584 generic.go:356] "Generic (PLEG): container finished" podID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerID="e8c74277bd601b37bf1df15d52bf272c22be0eb3cda7ae909d46d6aa5c215d4a" exitCode=2
Sep 18 19:57:44.437212 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:44.437056    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerDied","Data":"e8c74277bd601b37bf1df15d52bf272c22be0eb3cda7ae909d46d6aa5c215d4a"}
Sep 18 19:57:46.448989 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.448958    2584 generic.go:356] "Generic (PLEG): container finished" podID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerID="da27373e08cf2f5a13ac163fa194c372208c8ed19302cbe46135a1db9c85200c" exitCode=0
Sep 18 19:57:46.449306 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.449022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerDied","Data":"da27373e08cf2f5a13ac163fa194c372208c8ed19302cbe46135a1db9c85200c"}
Sep 18 19:57:46.564759 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.564736    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:57:46.700237 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.700170    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzvtw\" (UniqueName: \"kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw\") pod \"81e0676f-066f-4f4f-bfdb-aa9bed853758\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") "
Sep 18 19:57:46.700237 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.700222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") pod \"81e0676f-066f-4f4f-bfdb-aa9bed853758\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") "
Sep 18 19:57:46.700414 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.700275    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config\") pod \"81e0676f-066f-4f4f-bfdb-aa9bed853758\" (UID: \"81e0676f-066f-4f4f-bfdb-aa9bed853758\") "
Sep 18 19:57:46.700701 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.700676    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81e0676f-066f-4f4f-bfdb-aa9bed853758" (UID: "81e0676f-066f-4f4f-bfdb-aa9bed853758"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:46.702216 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.702194    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81e0676f-066f-4f4f-bfdb-aa9bed853758" (UID: "81e0676f-066f-4f4f-bfdb-aa9bed853758"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:46.702286 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.702259    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw" (OuterVolumeSpecName: "kube-api-access-bzvtw") pod "81e0676f-066f-4f4f-bfdb-aa9bed853758" (UID: "81e0676f-066f-4f4f-bfdb-aa9bed853758"). InnerVolumeSpecName "kube-api-access-bzvtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:46.801325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.801301    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e0676f-066f-4f4f-bfdb-aa9bed853758-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:57:46.801325 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.801325    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:57:46.801452 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:46.801334    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvtw\" (UniqueName: \"kubernetes.io/projected/81e0676f-066f-4f4f-bfdb-aa9bed853758-kube-api-access-bzvtw\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 19:57:47.455731 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.455706    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"
Sep 18 19:57:47.456100 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.455699    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s" event={"ID":"81e0676f-066f-4f4f-bfdb-aa9bed853758","Type":"ContainerDied","Data":"e10100e98080fe3eb446d2734eb9bc03f73dcd01af82b69f3fc7859a98f74e7b"}
Sep 18 19:57:47.456100 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.455880    2584 scope.go:117] "RemoveContainer" containerID="e8c74277bd601b37bf1df15d52bf272c22be0eb3cda7ae909d46d6aa5c215d4a"
Sep 18 19:57:47.465869 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.465849    2584 scope.go:117] "RemoveContainer" containerID="da27373e08cf2f5a13ac163fa194c372208c8ed19302cbe46135a1db9c85200c"
Sep 18 19:57:47.488735 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.488705    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:57:47.496280 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:47.496256    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa781-predictor-549d86d794-2q82s"]
Sep 18 19:57:48.510488 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:57:48.510453    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" path="/var/lib/kubelet/pods/81e0676f-066f-4f4f-bfdb-aa9bed853758/volumes"
Sep 18 19:59:18.570243 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:59:18.570138    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 19:59:18.575917 ip-10-0-140-94 kubenswrapper[2584]: I0918 19:59:18.575898    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 20:04:18.592964 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:04:18.592856    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 20:04:18.599722 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:04:18.599702    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 20:05:34.404229 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:34.404187    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 20:05:34.404812 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:34.404428    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container" containerID="cri-o://65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d" gracePeriod=30
Sep 18 20:05:34.404812 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:34.404487    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kube-rbac-proxy" containerID="cri-o://2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078" gracePeriod=30
Sep 18 20:05:35.239623 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:35.239582    2584 generic.go:356] "Generic (PLEG): container finished" podID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerID="2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078" exitCode=2
Sep 18 20:05:35.239802 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:35.239674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerDied","Data":"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"}
Sep 18 20:05:37.654451 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.654424    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 20:05:37.840143 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.840038    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nggzr\" (UniqueName: \"kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr\") pod \"9aaa8d21-598f-4f7d-8c21-d974969b3961\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") "
Sep 18 20:05:37.840143 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.840115    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config\") pod \"9aaa8d21-598f-4f7d-8c21-d974969b3961\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") "
Sep 18 20:05:37.840368 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.840188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") pod \"9aaa8d21-598f-4f7d-8c21-d974969b3961\" (UID: \"9aaa8d21-598f-4f7d-8c21-d974969b3961\") "
Sep 18 20:05:37.840577 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.840544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9aaa8d21-598f-4f7d-8c21-d974969b3961" (UID: "9aaa8d21-598f-4f7d-8c21-d974969b3961"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:37.842425 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.842401    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr" (OuterVolumeSpecName: "kube-api-access-nggzr") pod "9aaa8d21-598f-4f7d-8c21-d974969b3961" (UID: "9aaa8d21-598f-4f7d-8c21-d974969b3961"). InnerVolumeSpecName "kube-api-access-nggzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:37.842425 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.842410    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9aaa8d21-598f-4f7d-8c21-d974969b3961" (UID: "9aaa8d21-598f-4f7d-8c21-d974969b3961"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:37.941599 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.941560    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aaa8d21-598f-4f7d-8c21-d974969b3961-proxy-tls\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 20:05:37.941599 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.941599    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nggzr\" (UniqueName: \"kubernetes.io/projected/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-api-access-nggzr\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 20:05:37.941599 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:37.941613    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aaa8d21-598f-4f7d-8c21-d974969b3961-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-94.ec2.internal\" DevicePath \"\""
Sep 18 20:05:38.252782 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.252743    2584 generic.go:356] "Generic (PLEG): container finished" podID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerID="65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d" exitCode=0
Sep 18 20:05:38.252974 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.252801    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"
Sep 18 20:05:38.252974 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.252805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerDied","Data":"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"}
Sep 18 20:05:38.252974 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.252899    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc" event={"ID":"9aaa8d21-598f-4f7d-8c21-d974969b3961","Type":"ContainerDied","Data":"2466ca25b6d863e90c06ba08cba632ec79e0bfc297f8f4f4f41f0a34512cc32c"}
Sep 18 20:05:38.252974 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.252936    2584 scope.go:117] "RemoveContainer" containerID="2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"
Sep 18 20:05:38.261988 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.261970    2584 scope.go:117] "RemoveContainer" containerID="65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"
Sep 18 20:05:38.269128 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.269109    2584 scope.go:117] "RemoveContainer" containerID="2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"
Sep 18 20:05:38.269377 ip-10-0-140-94 kubenswrapper[2584]: E0918 20:05:38.269359    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078\": container with ID starting with 2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078 not found: ID does not exist" containerID="2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"
Sep 18 20:05:38.269441 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.269382    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078"} err="failed to get container status \"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078\": rpc error: code = NotFound desc = could not find container \"2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078\": container with ID starting with 2888c46ff104369047e84529f3293e9e807e548e0e2fc15092e9337716b1d078 not found: ID does not exist"
Sep 18 20:05:38.269441 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.269401    2584 scope.go:117] "RemoveContainer" containerID="65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"
Sep 18 20:05:38.269679 ip-10-0-140-94 kubenswrapper[2584]: E0918 20:05:38.269661    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d\": container with ID starting with 65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d not found: ID does not exist" containerID="65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"
Sep 18 20:05:38.269729 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.269685    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d"} err="failed to get container status \"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d\": rpc error: code = NotFound desc = could not find container \"65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d\": container with ID starting with 65b7b9f8c6ebcdd40b6f2c6486f5c63254d51ad65c0ca47152154f5e35b8bd8d not found: ID does not exist"
Sep 18 20:05:38.275118 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.275097    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 20:05:38.279270 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.279249    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-38355-predictor-64ffd94458-8hgfc"]
Sep 18 20:05:38.509885 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:05:38.509817    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" path="/var/lib/kubelet/pods/9aaa8d21-598f-4f7d-8c21-d974969b3961/volumes"
Sep 18 20:06:06.279366 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279333    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jg6d/must-gather-2xk5b"]
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279599    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279614    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279625    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279650    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279675    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279687    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279694    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279700    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279705    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279716    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279722    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279771    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279778    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279785    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aaa8d21-598f-4f7d-8c21-d974969b3961" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279792    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kube-rbac-proxy"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279797    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40d0190-4b0f-4097-ab43-d459fa5b2c8e" containerName="kserve-container"
Sep 18 20:06:06.279915 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.279802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e0676f-066f-4f4f-bfdb-aa9bed853758" containerName="kube-rbac-proxy"
Sep 18 20:06:06.282685 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.282669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.285250 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.285226    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jg6d\"/\"kube-root-ca.crt\""
Sep 18 20:06:06.285574 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.285554    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jg6d\"/\"openshift-service-ca.crt\""
Sep 18 20:06:06.287264 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.287247    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7jg6d\"/\"default-dockercfg-76422\""
Sep 18 20:06:06.295417 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.295399    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jg6d/must-gather-2xk5b"]
Sep 18 20:06:06.340958 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.340936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-must-gather-output\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.341056 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.340984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzk66\" (UniqueName: \"kubernetes.io/projected/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-kube-api-access-wzk66\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.441268 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.441234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzk66\" (UniqueName: \"kubernetes.io/projected/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-kube-api-access-wzk66\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.441376 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.441288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-must-gather-output\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.441659 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.441623    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-must-gather-output\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.450618 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.450600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzk66\" (UniqueName: \"kubernetes.io/projected/aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac-kube-api-access-wzk66\") pod \"must-gather-2xk5b\" (UID: \"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac\") " pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.591999 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.591941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jg6d/must-gather-2xk5b"
Sep 18 20:06:06.861506 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.861477    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jg6d/must-gather-2xk5b"]
Sep 18 20:06:06.863874 ip-10-0-140-94 kubenswrapper[2584]: W0918 20:06:06.863835    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeae8bbc_62f7_45ac_aeaf_47e90c9f4cac.slice/crio-4b1649c8c972da431b2ce0c4539c94531a0234cdacb122983d28670a56ae0006 WatchSource:0}: Error finding container 4b1649c8c972da431b2ce0c4539c94531a0234cdacb122983d28670a56ae0006: Status 404 returned error can't find the container with id 4b1649c8c972da431b2ce0c4539c94531a0234cdacb122983d28670a56ae0006
Sep 18 20:06:06.865676 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:06.865657    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:06:07.360573 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:07.360536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jg6d/must-gather-2xk5b" event={"ID":"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac","Type":"ContainerStarted","Data":"4b1649c8c972da431b2ce0c4539c94531a0234cdacb122983d28670a56ae0006"}
Sep 18 20:06:08.368603 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:08.368559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jg6d/must-gather-2xk5b" event={"ID":"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac","Type":"ContainerStarted","Data":"b9193045b06cb39eedcf8e38c31e10c5d434af0204e7cfd2846f68bcdd83064d"}
Sep 18 20:06:08.369025 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:08.368617    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jg6d/must-gather-2xk5b" event={"ID":"aeae8bbc-62f7-45ac-aeaf-47e90c9f4cac","Type":"ContainerStarted","Data":"4f19e33abe616936cc9e3261fd5f4b9924b98ea798a6ba3132945a9c0dd932e2"}
Sep 18 20:06:08.387956 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:08.387890    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jg6d/must-gather-2xk5b" podStartSLOduration=1.5984432480000001 podStartE2EDuration="2.387870668s" totalImagesPullingTime="789.42742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:08.38682112 +0000 UTC m=+3112.105478420" watchObservedRunningTime="2026-09-18 20:06:08.387870668 +0000 UTC m=+3112.106527971"
Sep 18 20:06:09.199079 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:09.199029    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7thst_f6e38969-99fd-4925-9313-6ef4ac0674c2/global-pull-secret-syncer/0.log"
Sep 18 20:06:09.393246 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:09.393203    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fs5tv_7ef57494-7501-4137-8770-70f06d6aa386/konnectivity-agent/0.log"
Sep 18 20:06:09.495215 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:09.495149    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-94.ec2.internal_bc46e81cf898f71be4cf1f3edbfc59bf/haproxy/0.log"
Sep 18 20:06:13.819989 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:13.819953    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-vrnxw_58a99a04-199f-409c-80c5-ecf84c2ba538/cluster-monitoring-operator/0.log"
Sep 18 20:06:14.087751 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:14.087660    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhgv7_d22ad3ab-1106-4f5c-9142-83bc73712779/node-exporter/0.log"
Sep 18 20:06:14.127036 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:14.126997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhgv7_d22ad3ab-1106-4f5c-9142-83bc73712779/kube-rbac-proxy/0.log"
Sep 18 20:06:14.155540 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:14.155426    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhgv7_d22ad3ab-1106-4f5c-9142-83bc73712779/init-textfile/0.log"
Sep 18 20:06:16.423709 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:16.423683    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-kx2c8_10a41341-2ad5-454e-8d6b-f954f8573c7a/networking-console-plugin/0.log"
Sep 18 20:06:16.932029 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:16.931996    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"]
Sep 18 20:06:16.935356 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:16.935324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:16.961716 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:16.961690    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"]
Sep 18 20:06:17.039804 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.039760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-lib-modules\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.039976 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.039831    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-sys\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.039976 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.039889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhm9\" (UniqueName: \"kubernetes.io/projected/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-kube-api-access-jqhm9\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.039976 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.039946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-proc\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.040122 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.040018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-podres\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141416 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-proc\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141416 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141419    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-podres\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-lib-modules\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-sys\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141502    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhm9\" (UniqueName: \"kubernetes.io/projected/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-kube-api-access-jqhm9\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-proc\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-podres\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141613    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-sys\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.141684 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.141656    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-lib-modules\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.168814 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.168763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhm9\" (UniqueName: \"kubernetes.io/projected/13d93cee-9cbc-44fb-ac0d-8dd74e6514eb-kube-api-access-jqhm9\") pod \"perf-node-gather-daemonset-t4bzq\" (UID: \"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb\") " pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.247239 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.247169    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:17.542350 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.542316    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"]
Sep 18 20:06:17.545002 ip-10-0-140-94 kubenswrapper[2584]: W0918 20:06:17.544963    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13d93cee_9cbc_44fb_ac0d_8dd74e6514eb.slice/crio-e627692e49698e1e49d68e301659cc9940f06e8009887ef6bff112882030a1b8 WatchSource:0}: Error finding container e627692e49698e1e49d68e301659cc9940f06e8009887ef6bff112882030a1b8: Status 404 returned error can't find the container with id e627692e49698e1e49d68e301659cc9940f06e8009887ef6bff112882030a1b8
Sep 18 20:06:17.745050 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:17.744995    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d4f9d9869-fdlk9_216ef592-2266-4e7c-a766-88e77e072609/console/0.log"
Sep 18 20:06:18.420225 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:18.420186    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq" event={"ID":"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb","Type":"ContainerStarted","Data":"c7143dcdfd51c0f08f76a62cc48b0878f9d4d23e8225ab330af3d36a580af6c6"}
Sep 18 20:06:18.420393 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:18.420284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq" event={"ID":"13d93cee-9cbc-44fb-ac0d-8dd74e6514eb","Type":"ContainerStarted","Data":"e627692e49698e1e49d68e301659cc9940f06e8009887ef6bff112882030a1b8"}
Sep 18 20:06:18.420393 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:18.420312    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:18.446103 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:18.446058    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq" podStartSLOduration=2.446043002 podStartE2EDuration="2.446043002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:18.444714719 +0000 UTC m=+3122.163372020" watchObservedRunningTime="2026-09-18 20:06:18.446043002 +0000 UTC m=+3122.164700352"
Sep 18 20:06:19.529841 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:19.529792    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flnrv_64df114d-0390-4d42-b107-656830b8a27c/dns/0.log"
Sep 18 20:06:19.571368 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:19.571330    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flnrv_64df114d-0390-4d42-b107-656830b8a27c/kube-rbac-proxy/0.log"
Sep 18 20:06:19.744696 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:19.744671    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jxxz_48a0cfb3-668d-4396-b407-133871aa0d41/dns-node-resolver/0.log"
Sep 18 20:06:20.262169 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:20.262128    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-588fcb9b4c-kgqck_9e1f8ab7-0985-4029-b609-acefa3c3fc35/registry/0.log"
Sep 18 20:06:20.285475 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:20.285456    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-62z5k_d2b55544-874b-4f30-baa8-9288cde4cf37/node-ca/0.log"
Sep 18 20:06:21.517922 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.517891    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pfs8t_09842818-d867-4448-b34b-cdded954cb09/serve-healthcheck-canary/0.log"
Sep 18 20:06:21.903100 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.902998    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-chdrd_2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8/insights-operator/0.log"
Sep 18 20:06:21.903669 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.903618    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-chdrd_2e2a79a7-d8c7-4dd3-b0b7-80e4add24cd8/insights-operator/1.log"
Sep 18 20:06:21.923481 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.923463    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rbnz8_e13f0853-dab1-403c-972f-0079739bd48a/kube-rbac-proxy/0.log"
Sep 18 20:06:21.948449 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.948428    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rbnz8_e13f0853-dab1-403c-972f-0079739bd48a/exporter/0.log"
Sep 18 20:06:21.971194 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:21.971171    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rbnz8_e13f0853-dab1-403c-972f-0079739bd48a/extractor/0.log"
Sep 18 20:06:24.434051 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:24.434022    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7jg6d/perf-node-gather-daemonset-t4bzq"
Sep 18 20:06:24.544276 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:24.544245    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-qvptg_c0f8df64-fe81-48ab-81a0-5a334fba2994/server/0.log"
Sep 18 20:06:24.644058 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:24.644024    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-5mfdb_0dad30e9-45b1-45bb-8f3a-2ffb22a98a13/manager/0.log"
Sep 18 20:06:24.742790 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:24.742724    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5df28_619592f3-8209-4af1-a1d4-241b95c43628/seaweedfs/0.log"
Sep 18 20:06:31.180627 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.180593    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:06:31.204912 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.204871    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/egress-router-binary-copy/0.log"
Sep 18 20:06:31.230294 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.230241    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/cni-plugins/0.log"
Sep 18 20:06:31.258522 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.258497    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/bond-cni-plugin/0.log"
Sep 18 20:06:31.290574 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.290556    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/routeoverride-cni/0.log"
Sep 18 20:06:31.321747 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.321728    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/whereabouts-cni-bincopy/0.log"
Sep 18 20:06:31.348953 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.348927    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsx2n_0f46f69a-eedc-4d12-9a53-96baae2780b6/whereabouts-cni/0.log"
Sep 18 20:06:31.740563 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.740536    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qzkxt_393f2e3b-3dbb-4d68-b145-ba860b2a0c08/kube-multus/0.log"
Sep 18 20:06:31.809149 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.809124    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9829_738fc24a-cf2d-4ccc-995e-eb9e54e6b044/network-metrics-daemon/0.log"
Sep 18 20:06:31.830797 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:31.830776    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9829_738fc24a-cf2d-4ccc-995e-eb9e54e6b044/kube-rbac-proxy/0.log"
Sep 18 20:06:33.143459 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.143432    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-controller/0.log"
Sep 18 20:06:33.172236 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.172213    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/1.log"
Sep 18 20:06:33.172236 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.172217    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovn-acl-logging/0.log"
Sep 18 20:06:33.199611 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.199589    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/kube-rbac-proxy-node/0.log"
Sep 18 20:06:33.228629 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.228592    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:06:33.252222 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.252196    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/northd/0.log"
Sep 18 20:06:33.278509 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.278492    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/nbdb/0.log"
Sep 18 20:06:33.305416 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.305399    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/sbdb/0.log"
Sep 18 20:06:33.359372 ip-10-0-140-94 kubenswrapper[2584]: I0918 20:06:33.359289    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5x2c_1d130a7b-ffca-47cf-9c74-7e4632c70b4a/ovnkube-controller/0.log"