Sep 17 16:39:50.167051 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 16:39:50.167063 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 16:39:50.167070 ip-10-0-143-112 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 16:39:50.167653 ip-10-0-143-112 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 16:40:00.208207 ip-10-0-143-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 16:40:00.208229 ip-10-0-143-112 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f31ed55d9f0d477b939f7b7522e907e1 --
Sep 17 16:41:45.894171 ip-10-0-143-112 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:41:49.204552 ip-10-0-143-112 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:49.204552 ip-10-0-143-112 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:41:49.204552 ip-10-0-143-112 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:49.204552 ip-10-0-143-112 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:41:49.204552 ip-10-0-143-112 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:41:49.205279 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.204626    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:41:49.208390 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208374    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.208390 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208391    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208394    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208398    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208401    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208405    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208408    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208411    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208413    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208416    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208420    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208423    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208426    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208429    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208431    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208434    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208436    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208439    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208441    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208444    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.208452 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208447    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208449    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208452    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208455    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208457    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208460    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208463    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208468    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208472    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208476    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208480    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208484    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208487    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208490    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208493    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208496    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208499    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208502    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208504    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.208898 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208506    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208509    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208512    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208514    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208516    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208519    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208521    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208524    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208526    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208528    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208531    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208533    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208536    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208538    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208541    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208544    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208548    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208551    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208554    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208556    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.209401 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208559    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208561    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208564    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208566    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208570    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208572    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208575    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208578    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208580    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208583    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208586    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208589    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208592    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208594    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208597    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208599    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208601    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208604    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208608    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.209897 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208611    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208614    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208616    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208619    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208622    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208624    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208627    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208629    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208633    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208635    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208638    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208640    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208643    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208645    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208648    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208650    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208653    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208655    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208658    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208660    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.210370 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.208662    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210003    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210012    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210015    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210021    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210025    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210028    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210031    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210034    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210037    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210040    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210042    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210045    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210047    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210050    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210052    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210055    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210058    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210060    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210063    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.210842 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210066    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210068    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210071    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210073    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210076    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210079    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210081    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210084    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210086    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210088    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210100    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210103    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210106    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210108    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210112    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210116    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210119    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210123    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210126    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210129    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.211356 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210132    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210134    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210138    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210140    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210143    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210145    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210148    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210150    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210153    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210155    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210158    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210161    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210164    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210167    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210169    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210171    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210174    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210176    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210179    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210181    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.211835 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210184    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210186    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210189    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210191    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210194    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210196    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210198    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210201    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210204    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210208    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210212    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210215    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210217    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210220    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210222    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210225    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210227    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210230    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210233    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.212329 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210235    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210238    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210241    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210244    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210247    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210249    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210252    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210254    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210257    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210259    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210262    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210264    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210267    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210270    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210274    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210277    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210280    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210283    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210285    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210288    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.212795 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210291    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210675    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210681    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210690    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210693    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210697    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210699    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210702    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210705    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210708    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210711    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210713    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210716    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210718    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210721    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210724    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210726    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210729    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210732    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.213285 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210735    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210739    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210742    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210745    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210748    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210751    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210753    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210756    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210758    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210761    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210763    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210766    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210768    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210771    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210773    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210775    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210778    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210788    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210790    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.213840 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210793    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210795    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210798    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210800    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210802    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210805    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210807    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210809    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210812    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210815    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210817    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210820    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210822    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210824    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210827    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210829    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210832    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210834    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210837    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210839    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210844    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.214338 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210847    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210849    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210852    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210854    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210857    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210859    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210862    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210864    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210866    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210869    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210877    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210879    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210881    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210884    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210886    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210889    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210891    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210893    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210896    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210901    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.214843 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210919    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210922    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210925    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210928    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210931    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210934    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210936    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210939    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210942    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210945    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210947    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210951    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210953    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210956    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210960    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210963    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210966    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210968    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210971    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.215340 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210974    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.210976    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211043    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211075    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211083    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211087    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211091    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211094    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211098    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211103    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211106    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211109    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211112    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211116    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211119    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211122    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211125    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211128    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211130    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211133    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211137    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211140    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211143    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211145    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:41:49.215813 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211149    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211152    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211155    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211158    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211161    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211164    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211167    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211170    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211173    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211178    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211184    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211187    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211190    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211194    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211197    2580 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211200    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211204    2580 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211207    2580 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211210    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211213    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211215    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211219    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211221    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211224    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211227    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:41:49.216399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211230    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211232    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211235    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211238    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211240    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211243    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211246    2580 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211249    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211252    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211255    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211258    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211261    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211264    2580 flags.go:64] FLAG: --help="false"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211267    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211270    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211272    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211275    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211278    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211281    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211284    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211286    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211289    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211292    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:41:49.217026 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211295    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211298    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211301    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211303    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211306    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211309    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211311    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211314    2580 flags.go:64] FLAG: --lock-file=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211316    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211320    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211323    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211333    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211336    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211338    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211341    2580 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211344    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211347    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211350    2580 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211352    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211356    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211359    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211363    2580 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211366    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211369    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211372    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:41:49.217646 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211374    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211377    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211380    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211383    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211390    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211393    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211396    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211399    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211404    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211409    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211412    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211414    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211417    2580 flags.go:64] FLAG: --port="10250"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211420    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211423    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00b62b223536a2c1e"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211426    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211429    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211432    2580 flags.go:64] FLAG: --register-node="true"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211435    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211438    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211441    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211443    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211446    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211450    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211452    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:41:49.218276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211455    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211458    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211461    2580 flags.go:64] FLAG: --runonce="false"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211463    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211466    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211469    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211472    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211474    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211477    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211481    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211485    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211487    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211490    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211493    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211501    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211506    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211508    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211511    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211516    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211519    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211521    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211525    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211528    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211531    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211533    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:41:49.218872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211536    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211539    2580 flags.go:64] FLAG: --v="2"
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211546    2580 flags.go:64] FLAG: --version="false"
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211550    2580 flags.go:64] FLAG: --vmodule=""
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211553    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.211557    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211691    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211695    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211698    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211701    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211705    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211709    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211712    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211714    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211717    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211720    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211724    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211726    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211729    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211732    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.219482 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211735    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211738    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211742    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211745    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211748    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211751    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211762    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211765    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211768    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211770    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211773    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211776    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211778    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211781    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211783    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211786    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211789    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211792    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211794    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.219979 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211797    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211800    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211802    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211805    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211807    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211810    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211812    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211815    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211817    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211821    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211824    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211826    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211829    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211831    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211834    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211838    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211842    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211844    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211847    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211850    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.220693 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211852    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211855    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211857    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211860    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211862    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211866    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211868    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211871    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211873    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211876    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211878    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211881    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211883    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211886    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211889    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211891    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211894    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211896    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211899    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211915    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.221324 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211919    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211923    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211926    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211929    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211931    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211934    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211936    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211940    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211943    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211946    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211949    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211951    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211954    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211956    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211959    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211963    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211966    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211969    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211972    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.221828 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211974    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211977    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211980    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211982    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211985    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211988    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.211990    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.212549    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.220941    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.220959    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.220986    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.221000    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221293    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221301    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.222301 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221305    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221308    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221312    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221315    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221318    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221320    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221324    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221327    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221329    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221332    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221335    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221338    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221341    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221344    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221347    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221350    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221352    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221355    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221358    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221360    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.222649 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221363    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221365    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221368    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221370    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221373    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221375    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221378    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221382    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221385    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221388    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221390    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221393    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221396    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221400    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221404    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221407    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221410    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221413    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221415    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.223184 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221418    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221421    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221423    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221425    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221430    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221433    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221437    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221442    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221444    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221447    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221450    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221452    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221455    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221458    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221460    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221463    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221465    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221468    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221470    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221473    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.223636 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221476    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221479    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221481    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221484    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221488    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221491    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221494    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221498    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221501    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221504    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221507    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221510    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221512    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221516    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221519    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221522    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221525    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221527    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221530    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.224131 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221533    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221535    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221538    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221540    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221543    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221546    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221548    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221551    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221553    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221556    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221558    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221561    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221563    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221567    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221570    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221572    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221575    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221577    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.224583 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221580    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.221585    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221724    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221730    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221733    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221736    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221739    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221742    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221745    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221748    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221751    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221753    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221756    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221759    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221765    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:41:49.225027 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221768    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221771    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221773    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221776    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221778    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221781    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221784    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221786    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221789    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221791    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221794    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221796    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221799    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221813    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221816    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221819    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221823    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221827    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:41:49.225389 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221834    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221837    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221840    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221843    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221845    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221848    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221850    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221853    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221855    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221858    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221861    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221863    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221866    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221869    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221873    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221876    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221880    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221882    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221885    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:41:49.225861 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221887    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221890    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221892    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221895    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221897    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221900    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221919    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221924    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221927    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221930    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221934    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221937    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221939    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221942    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221944    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221947    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221949    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221952    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221954    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221956    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:41:49.226355 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221959    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221961    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221964    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221966    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221969    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221971    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221974    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221977    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221980    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221982    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221985    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221987    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221989    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221992    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221994    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221997    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.221999    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222001    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222004    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222007    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:41:49.226844 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222010    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222012    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222014    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222017    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222019    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222022    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222024    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:49.222027    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.222031    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.222133    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.224376    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.225181    2580 server.go:1019] "Starting client certificate rotation"
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.225280    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:49.227342 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.225312    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:41:49.227705 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.227683    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:49.228790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.228773    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:41:49.232665 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.232649    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:41:49.234407 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.234394    2580 log.go:25] "Validated CRI v1 image API"
Sep 17 16:41:49.235278 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.235248    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:41:49.237607 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.237581    2580 fs.go:135] Filesystem UUIDs: map[0ff0f8f5-a17a-4c4a-b613-729717b675d5:/dev/nvme0n1p3 65c4d998-1e0f-4512-afb3-924887e35b85:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 16:41:49.237675 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.237607    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:41:49.242944 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.242822    2580 manager.go:216] Machine: {Timestamp:2026-09-17 16:41:49.242363966 +0000 UTC m=+1.939866154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100743 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec220fee31fbb56faa88e30977453523 SystemUUID:ec220fee-31fb-b56f-aa88-e30977453523 BootID:f31ed55d-9f0d-477b-939f-7b7522e907e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:1f:55:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:1f:55:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ef:a5:c8:12:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:41:49.242944 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.242941    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:41:49.243051 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.243011    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:41:49.244531 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.244511    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:41:49.244652 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.244534    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-112.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:41:49.245750 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.245739    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:41:49.245790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.245753    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:41:49.245790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.245760    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:49.245790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.245769    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:41:49.247252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.247241    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:41:49.251318 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251307    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:49.251439 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251430    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:41:49.251474 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251465    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:41:49.251514 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251493    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:41:49.251542 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251517    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:41:49.251542 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251531    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:41:49.251593 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.251541    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:41:49.252297 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.252282    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:49.252360 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.252308    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:41:49.264289 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.264271    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:41:49.265305 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.265268    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:41:49.267058 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267042    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:41:49.267116 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267073    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:41:49.267215 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267205    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:41:49.267245 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267220    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:41:49.267245 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267226    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:41:49.267245 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267232    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:41:49.267245 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267240    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:41:49.267245 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267245    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:41:49.267892 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267879    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:41:49.267892 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267891    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:41:49.267892 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267898    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:41:49.268034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267920    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:41:49.268034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267931    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:41:49.268034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267939    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:41:49.268034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267960    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:41:49.268034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.267965    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:41:49.269435 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.269409    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:49.271822 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.271806    2580 server.go:1268] "Started kubelet"
Sep 17 16:41:49.271941 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.271845    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:41:49.272015 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.271926    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:41:49.272015 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.271986    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:41:49.272311 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.272294    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:41:49.272576 ip-10-0-143-112 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:41:49.273781 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.273769    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:41:49.274237 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.274207    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:41:49.274690 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.274671    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-112.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 16:41:49.277638 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.277620    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:41:49.280998 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.280982    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:41:49.281092 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.280999    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:49.281092 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.281058    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.281647 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281622    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:41:49.281647 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281641    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:41:49.281647 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281622    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:41:49.281818 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281743    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:41:49.281818 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281752    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:41:49.281900 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281891    2580 factory.go:55] Registering systemd factory
Sep 17 16:41:49.281967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.281924    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:41:49.282241 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282226    2580 factory.go:153] Registering CRI-O factory
Sep 17 16:41:49.282241 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282244    2580 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:41:49.282383 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282293    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:41:49.282383 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282313    2580 factory.go:103] Registering Raw factory
Sep 17 16:41:49.282383 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282327    2580 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:41:49.282648 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.282635    2580 manager.go:349] Starting recovery of all containers
Sep 17 16:41:49.284714 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.284693    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 16:41:49.284797 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.284749    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-112.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 16:41:49.301453 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.301421    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phgzv"
Sep 17 16:41:49.303698 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.303675    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phgzv"
Sep 17 16:41:49.307490 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.307469    2580 manager.go:354] Recovery completed
Sep 17 16:41:49.311543 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.311529    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.313318 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.313301    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.313380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.313332    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.313380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.313343    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.314458 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.314445    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:41:49.314458 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.314459    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:41:49.314564 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.314478    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:41:49.314564 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.314482    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.315244 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.315232    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.315279 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.315256    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.315279 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.315265    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.316529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.316517    2580 policy_none.go:49] "None policy: Start"
Sep 17 16:41:49.316573 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.316533    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:41:49.316573 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.316543    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:41:49.318547 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.318537    2580 policy_none.go:47] "Start"
Sep 17 16:41:49.350479 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350463    2580 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.350499    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350518    2580 server.go:87] "Starting device plugin registration server"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350699    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350716    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350799    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350863    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.350872    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.352084    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.352123    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.356249    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.357478    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.357518    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.357538    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.357578    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 16:41:49.365431 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.360263    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:49.451638 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.451611    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.452521 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.452506    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.452599 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.452537    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.452599 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.452549    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.452599 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.452575    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.458479 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.458437    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"]
Sep 17 16:41:49.458527 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.458497    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.459270 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.459255    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.459270 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.459281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.459418 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.459292    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.461333 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.461319    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.461396 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.461340    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-112.ec2.internal\": node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.461713 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.461701    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.461844 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.461828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.461920 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.461868    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.462376 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462355    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.462452 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462385    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.462452 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462394    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.462537 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462478    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.462537 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462514    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.462537 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.462527    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.464874 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.464857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.464967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.464918    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:41:49.465662 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.465645    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:41:49.465737 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.465675    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:41:49.465737 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.465688    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:41:49.479456 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.479437    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-112.ec2.internal\" not found" node="ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.482070 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.482050    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.483448 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.483428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.483547 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.483452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.483547 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.483484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/607b69b0e690d92464a627a4aea229b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"607b69b0e690d92464a627a4aea229b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.483547 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.483499    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-112.ec2.internal\" not found" node="ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.583011 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.582984    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.584096 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.584164 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.584164 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/607b69b0e690d92464a627a4aea229b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"607b69b0e690d92464a627a4aea229b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.584246 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/607b69b0e690d92464a627a4aea229b6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-112.ec2.internal\" (UID: \"607b69b0e690d92464a627a4aea229b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.584246 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.584246 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.584188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c26dd9f0a085e62835ad42e7cd0728a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal\" (UID: \"0c26dd9f0a085e62835ad42e7cd0728a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.683899 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.683860    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.783636 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.783612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.784692 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.784674    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.790168 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:49.789506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:49.885747 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.885572    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:49.986101 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:49.986061    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.086636 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.086572    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.187034 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.187006    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.225249 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.225226    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:41:50.225666 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.225357    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:41:50.287728 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.287701    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.309008 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.308976    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:36:49 +0000 UTC" deadline="2028-07-02 23:26:03.802665542 +0000 UTC"
Sep 17 16:41:50.309008 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.309004    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15702h44m13.493665577s"
Sep 17 16:41:50.387821 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.387798    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.432237 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.432210    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:50.479165 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.479142    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:50.487920 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.487886    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.568448 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:50.568421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c26dd9f0a085e62835ad42e7cd0728a.slice/crio-27af74293e8bf206f83a68af171a844797bbefc74b8dbc616d90326ff7a4fe00 WatchSource:0}: Error finding container 27af74293e8bf206f83a68af171a844797bbefc74b8dbc616d90326ff7a4fe00: Status 404 returned error can't find the container with id 27af74293e8bf206f83a68af171a844797bbefc74b8dbc616d90326ff7a4fe00
Sep 17 16:41:50.569151 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:50.569132    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607b69b0e690d92464a627a4aea229b6.slice/crio-0e03b697449025469d0f9fd02a566a70b0c02e922a3793e61ec0ae97c6bce00c WatchSource:0}: Error finding container 0e03b697449025469d0f9fd02a566a70b0c02e922a3793e61ec0ae97c6bce00c: Status 404 returned error can't find the container with id 0e03b697449025469d0f9fd02a566a70b0c02e922a3793e61ec0ae97c6bce00c
Sep 17 16:41:50.573895 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.573883    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:41:50.588343 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.588314    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.688892 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.688820    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.789587 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:50.789562    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-112.ec2.internal\" not found"
Sep 17 16:41:50.838626 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.838603    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:50.882256 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.882230    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:50.892854 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.892834    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:50.893889 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.893877    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal"
Sep 17 16:41:50.902258 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:50.902242    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:41:51.253000 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.252947    2580 apiserver.go:52] "Watching apiserver"
Sep 17 16:41:51.260107 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.260074    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:41:51.260474 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.260454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal","openshift-network-operator/iptables-alerter-wrxq6","openshift-ovn-kubernetes/ovnkube-node-v4plf","kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal","openshift-dns/node-resolver-z24x4","openshift-multus/multus-additional-cni-plugins-kx7rd","openshift-multus/multus-c7z4p","openshift-multus/network-metrics-daemon-k8mwq","openshift-network-diagnostics/network-check-target-jznpc","kube-system/global-pull-secret-syncer-rzlws","kube-system/konnectivity-agent-xqzcq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf","openshift-cluster-node-tuning-operator/tuned-wjl99","openshift-image-registry/node-ca-4cz9d"]
Sep 17 16:41:51.263019 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.262998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.263118 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.263076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:41:51.265797 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.265627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.267782 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.267732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-chq5f\""
Sep 17 16:41:51.267876 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.267823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.268006 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.267985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.268302 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.267990    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.270019 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.269688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.270455 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.270165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvkx2\""
Sep 17 16:41:51.270455 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.270313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:41:51.270455 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.270336    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:41:51.270879 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.270860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:41:51.271321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.271302    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.271399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.271362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:41:51.271824 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.271790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.273471 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.273443    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.273884 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.273866    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:41:51.273983 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.273964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k5rpc\""
Sep 17 16:41:51.274220 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.274203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.277313 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.277297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.279171 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.279136    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.279501 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.279484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.280085 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.280066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f4w97\""
Sep 17 16:41:51.280283 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.280132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.282158 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282139    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:41:51.282398 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282381    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjwzk\""
Sep 17 16:41:51.282533 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282520    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.282688 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282646    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:41:51.282775 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282757    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.282839 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:41:51.284025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.282866    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:41:51.284025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.283032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:51.284025 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.283087    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:41:51.284901 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.284887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.286696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.286675    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:41:51.287066 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.287048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qtpqq\""
Sep 17 16:41:51.287300 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.287280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.289290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.289249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.289507 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.289404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v22nc\""
Sep 17 16:41:51.289604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.289542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:41:51.289665 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.289645    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.290261 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.290245    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.290457 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.290431    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:41:51.292880 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.292860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.294021 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-run\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294097 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-systemd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294097 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294190 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-env-overrides\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294190 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcthg\" (UniqueName: \"kubernetes.io/projected/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-kube-api-access-qcthg\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.294190 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6e802b9-240f-4828-b1af-e66cc459d335-hosts-file\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.294190 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-kubernetes\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-var-lib-kubelet\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-etc-tuned\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-node-log\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294321 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c4n\" (UniqueName: \"kubernetes.io/projected/d6e802b9-240f-4828-b1af-e66cc459d335-kube-api-access-h4c4n\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-systemd\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-sys\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-tmp\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-etc-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-ovn\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-serviceca\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.294529 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-lib-modules\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-slash\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-host\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-kubelet\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.294774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-config\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e802b9-240f-4828-b1af-e66cc459d335-tmp-dir\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhg8\" (UniqueName: \"kubernetes.io/projected/81fac588-8726-472a-8b17-249a1384ddbc-kube-api-access-nnhg8\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qs5s\" (UniqueName: \"kubernetes.io/projected/fc823426-4ef1-4d6e-86be-aff70a072c07-kube-api-access-6qs5s\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-kubelet-config\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-netns\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-bin\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8617572-e20f-4969-ac9d-b6e375107c96-ovn-node-metrics-cert\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.295070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.294937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc6j\" (UniqueName: \"kubernetes.io/projected/f8617572-e20f-4969-ac9d-b6e375107c96-kube-api-access-hhc6j\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.295581 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.295562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-cnibin\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.295634 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.295578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.295634 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.295596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-log-socket\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.296121 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.296095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-systemd-units\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.296220 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.296163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-script-lib\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.296220 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.296200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-modprobe-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.296312 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.296248    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-dbus\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.296312 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.296277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-var-lib-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.297577 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.297552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:41:51.298110 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.298054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7mcfc\""
Sep 17 16:41:51.298604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.298584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:41:51.298814 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.298798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:41:51.299030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.299015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:41:51.299175 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.299156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fj9nw\""
Sep 17 16:41:51.299320 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.299305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:41:51.299725 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.299705    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:41:51.300031 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.300223 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-netd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.300305 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-host\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.300351 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:51.300452 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysconfig\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.300527 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.300469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-conf\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.315693 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.315669    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8hzrr"
Sep 17 16:41:51.321776 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.321710    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8hzrr"
Sep 17 16:41:51.363552 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.363506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" event={"ID":"607b69b0e690d92464a627a4aea229b6","Type":"ContainerStarted","Data":"0e03b697449025469d0f9fd02a566a70b0c02e922a3793e61ec0ae97c6bce00c"}
Sep 17 16:41:51.365144 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.365121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"0c26dd9f0a085e62835ad42e7cd0728a","Type":"ContainerStarted","Data":"27af74293e8bf206f83a68af171a844797bbefc74b8dbc616d90326ff7a4fe00"}
Sep 17 16:41:51.383322 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.383303    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:41:51.400851 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhg8\" (UniqueName: \"kubernetes.io/projected/81fac588-8726-472a-8b17-249a1384ddbc-kube-api-access-nnhg8\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs5s\" (UniqueName: \"kubernetes.io/projected/fc823426-4ef1-4d6e-86be-aff70a072c07-kube-api-access-6qs5s\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-kubelet-config\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-bin\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8617572-e20f-4969-ac9d-b6e375107c96-ovn-node-metrics-cert\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.400979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-cnibin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbdk\" (UniqueName: \"kubernetes.io/projected/17ec5e11-8ebd-4267-bc43-f147545d2b58-kube-api-access-wlbdk\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.401017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-kubelet-config\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-modprobe-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-var-lib-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-host\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-bin\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtrp\" (UniqueName: \"kubernetes.io/projected/9336b6a4-be76-4751-92fa-9372275f0ddd-kube-api-access-8xtrp\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-system-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-socket-dir-parent\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-var-lib-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2097712-860e-4e0a-85ef-f699c9d02b3f-konnectivity-ca\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-sys-fs\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-host\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-modprobe-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-systemd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.401379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-systemd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcthg\" (UniqueName: \"kubernetes.io/projected/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-kube-api-access-qcthg\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401390    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-systemd\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-sys\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-systemd\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-bin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-multus-certs\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-sys\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ec5e11-8ebd-4267-bc43-f147545d2b58-host-slash\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2097712-860e-4e0a-85ef-f699c9d02b3f-agent-certs\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-device-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-ovn\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-serviceca\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-multus\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-daemon-config\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.402336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-run-ovn\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17ec5e11-8ebd-4267-bc43-f147545d2b58-iptables-alerter-script\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-slash\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-registration-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-slash\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-kubelet\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-config\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.401991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e802b9-240f-4828-b1af-e66cc459d335-tmp-dir\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-serviceca\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-k8s-cni-cncf-io\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-kubelet\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-netns\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-netns\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc6j\" (UniqueName: \"kubernetes.io/projected/f8617572-e20f-4969-ac9d-b6e375107c96-kube-api-access-hhc6j\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-cnibin\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.403201 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-cni-binary-copy\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-netns\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-log-socket\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-systemd-units\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-cnibin\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-script-lib\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-dbus\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e802b9-240f-4828-b1af-e66cc459d335-tmp-dir\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-conf-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-netd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysconfig\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-conf\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-run\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-var-lib-kubelet\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-kubelet\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.404154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-config\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-env-overrides\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6e802b9-240f-4828-b1af-e66cc459d335-hosts-file\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-kubernetes\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-etc-tuned\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-tmp\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-dbus\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-etc-kubernetes\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-ovnkube-script-lib\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-node-log\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c4n\" (UniqueName: \"kubernetes.io/projected/d6e802b9-240f-4828-b1af-e66cc459d335-kube-api-access-h4c4n\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-kubernetes\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.405002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-node-log\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-hostroot\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6e802b9-240f-4828-b1af-e66cc459d335-hosts-file\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.402997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-d\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-run\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-log-socket\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.403107    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f8617572-e20f-4969-ac9d-b6e375107c96-env-overrides\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-systemd-units\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-var-lib-kubelet\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.403176    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:51.903155933 +0000 UTC m=+4.600658132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysconfig\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-etc-sysctl-conf\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-etc-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-cni-netd\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.405877 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-lib-modules\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-system-cni-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crx66\" (UniqueName: \"kubernetes.io/projected/40ca1de2-f941-4a8f-80b7-b413cf381930-kube-api-access-crx66\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f8617572-e20f-4969-ac9d-b6e375107c96-etc-openvswitch\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-lib-modules\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-socket-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-host\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hbj\" (UniqueName: \"kubernetes.io/projected/42053039-27d2-43e0-80a1-22148a863095-kube-api-access-c4hbj\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81fac588-8726-472a-8b17-249a1384ddbc-host\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc823426-4ef1-4d6e-86be-aff70a072c07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.403983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc823426-4ef1-4d6e-86be-aff70a072c07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.405929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-etc-tuned\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.406019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81fac588-8726-472a-8b17-249a1384ddbc-tmp\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.406557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.406035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8617572-e20f-4969-ac9d-b6e375107c96-ovn-node-metrics-cert\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.409138 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.409119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcthg\" (UniqueName: \"kubernetes.io/projected/db51668a-9f53-4ee9-b8c3-1d0a9aaf826f-kube-api-access-qcthg\") pod \"node-ca-4cz9d\" (UID: \"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f\") " pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.409232 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.409216    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:51.409288 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.409239    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:51.409288 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.409261    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:51.409395 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.409343    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:51.909324941 +0000 UTC m=+4.606827136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:51.409537 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.409505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs5s\" (UniqueName: \"kubernetes.io/projected/fc823426-4ef1-4d6e-86be-aff70a072c07-kube-api-access-6qs5s\") pod \"multus-additional-cni-plugins-kx7rd\" (UID: \"fc823426-4ef1-4d6e-86be-aff70a072c07\") " pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.410531 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.410484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc6j\" (UniqueName: \"kubernetes.io/projected/f8617572-e20f-4969-ac9d-b6e375107c96-kube-api-access-hhc6j\") pod \"ovnkube-node-v4plf\" (UID: \"f8617572-e20f-4969-ac9d-b6e375107c96\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.410726 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.410704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhg8\" (UniqueName: \"kubernetes.io/projected/81fac588-8726-472a-8b17-249a1384ddbc-kube-api-access-nnhg8\") pod \"tuned-wjl99\" (UID: \"81fac588-8726-472a-8b17-249a1384ddbc\") " pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.411706 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.411687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c4n\" (UniqueName: \"kubernetes.io/projected/d6e802b9-240f-4828-b1af-e66cc459d335-kube-api-access-h4c4n\") pod \"node-resolver-z24x4\" (UID: \"d6e802b9-240f-4828-b1af-e66cc459d335\") " pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.504231 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-etc-kubernetes\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504231 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-hostroot\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504231 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crx66\" (UniqueName: \"kubernetes.io/projected/40ca1de2-f941-4a8f-80b7-b413cf381930-kube-api-access-crx66\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-socket-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hbj\" (UniqueName: \"kubernetes.io/projected/42053039-27d2-43e0-80a1-22148a863095-kube-api-access-c4hbj\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-hostroot\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-cnibin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbdk\" (UniqueName: \"kubernetes.io/projected/17ec5e11-8ebd-4267-bc43-f147545d2b58-kube-api-access-wlbdk\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-cnibin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtrp\" (UniqueName: \"kubernetes.io/projected/9336b6a4-be76-4751-92fa-9372275f0ddd-kube-api-access-8xtrp\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-system-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-socket-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-socket-dir-parent\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.504493 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2097712-860e-4e0a-85ef-f699c9d02b3f-konnectivity-ca\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-sys-fs\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-bin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-multus-certs\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ec5e11-8ebd-4267-bc43-f147545d2b58-host-slash\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2097712-860e-4e0a-85ef-f699c9d02b3f-agent-certs\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-device-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-multus\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-daemon-config\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-sys-fs\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-etc-kubernetes\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17ec5e11-8ebd-4267-bc43-f147545d2b58-iptables-alerter-script\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-socket-dir-parent\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-registration-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.504949    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:51.505193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-k8s-cni-cncf-io\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.504978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-netns\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.505012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:41:52.00499341 +0000 UTC m=+4.702495600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-netns\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-cni-binary-copy\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-conf-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-system-cni-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-kubelet\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-conf-dir\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17ec5e11-8ebd-4267-bc43-f147545d2b58-iptables-alerter-script\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-bin\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-multus-certs\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ec5e11-8ebd-4267-bc43-f147545d2b58-host-slash\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2097712-860e-4e0a-85ef-f699c9d02b3f-konnectivity-ca\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-device-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42053039-27d2-43e0-80a1-22148a863095-registration-dir\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.506069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-kubelet\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506540 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-run-k8s-cni-cncf-io\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506540 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.505926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40ca1de2-f941-4a8f-80b7-b413cf381930-host-var-lib-cni-multus\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506540 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.506014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-multus-daemon-config\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506540 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.506022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40ca1de2-f941-4a8f-80b7-b413cf381930-cni-binary-copy\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.506862 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.506846    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:41:51.507682 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.507663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2097712-860e-4e0a-85ef-f699c9d02b3f-agent-certs\") pod \"konnectivity-agent-xqzcq\" (UID: \"c2097712-860e-4e0a-85ef-f699c9d02b3f\") " pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.513857 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.513838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hbj\" (UniqueName: \"kubernetes.io/projected/42053039-27d2-43e0-80a1-22148a863095-kube-api-access-c4hbj\") pod \"aws-ebs-csi-driver-node-cdwnf\" (UID: \"42053039-27d2-43e0-80a1-22148a863095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.514253 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.514221    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtrp\" (UniqueName: \"kubernetes.io/projected/9336b6a4-be76-4751-92fa-9372275f0ddd-kube-api-access-8xtrp\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:51.514435 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.514415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbdk\" (UniqueName: \"kubernetes.io/projected/17ec5e11-8ebd-4267-bc43-f147545d2b58-kube-api-access-wlbdk\") pod \"iptables-alerter-wrxq6\" (UID: \"17ec5e11-8ebd-4267-bc43-f147545d2b58\") " pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.514506 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.514477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crx66\" (UniqueName: \"kubernetes.io/projected/40ca1de2-f941-4a8f-80b7-b413cf381930-kube-api-access-crx66\") pod \"multus-c7z4p\" (UID: \"40ca1de2-f941-4a8f-80b7-b413cf381930\") " pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.578491 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.578431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wjl99"
Sep 17 16:41:51.587054 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.587029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:41:51.601611 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.601584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cz9d"
Sep 17 16:41:51.602691 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.602665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z24x4"
Sep 17 16:41:51.611181 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.611154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx7rd"
Sep 17 16:41:51.619287 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.619266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c7z4p"
Sep 17 16:41:51.627947 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.627930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrxq6"
Sep 17 16:41:51.635605 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.635587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:41:51.646180 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.646161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf"
Sep 17 16:41:51.907555 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:51.907512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:51.907726 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.907679    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:51.907775 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:51.907752    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:52.907733626 +0000 UTC m=+5.605235802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:52.008531 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:52.008495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:52.008531 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:52.008547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:52.008780 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008674    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:52.008780 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008693    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:52.008780 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008706    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:52.008780 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008764    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:52.008780 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008774    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:53.008754291 +0000 UTC m=+5.706256492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:52.009040 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.008835    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:41:53.008817792 +0000 UTC m=+5.706319969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:52.325000 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:52.324957    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:51 +0000 UTC" deadline="2028-04-04 01:49:02.772773023 +0000 UTC"
Sep 17 16:41:52.325000 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:52.324986    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13545h7m10.447789927s"
Sep 17 16:41:52.419096 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.419066    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb51668a_9f53_4ee9_b8c3_1d0a9aaf826f.slice/crio-53095c2195d628c741cc60e503f7acdddd23d9dd365946f780e143d0ac2c4295 WatchSource:0}: Error finding container 53095c2195d628c741cc60e503f7acdddd23d9dd365946f780e143d0ac2c4295: Status 404 returned error can't find the container with id 53095c2195d628c741cc60e503f7acdddd23d9dd365946f780e143d0ac2c4295
Sep 17 16:41:52.421547 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.421522    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ca1de2_f941_4a8f_80b7_b413cf381930.slice/crio-d62900b23258e9f3238f0022747135c6ab41d4adc1d3c80cf16614c912546311 WatchSource:0}: Error finding container d62900b23258e9f3238f0022747135c6ab41d4adc1d3c80cf16614c912546311: Status 404 returned error can't find the container with id d62900b23258e9f3238f0022747135c6ab41d4adc1d3c80cf16614c912546311
Sep 17 16:41:52.448957 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.448934    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42053039_27d2_43e0_80a1_22148a863095.slice/crio-f4ea2c2fa20b886d1f59715f360fcb86ff05d4a0cbf214c623faf863a215b555 WatchSource:0}: Error finding container f4ea2c2fa20b886d1f59715f360fcb86ff05d4a0cbf214c623faf863a215b555: Status 404 returned error can't find the container with id f4ea2c2fa20b886d1f59715f360fcb86ff05d4a0cbf214c623faf863a215b555
Sep 17 16:41:52.449586 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.449557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc823426_4ef1_4d6e_86be_aff70a072c07.slice/crio-11bcb511642b0abcf528ee5d32a703d97ce1e54e897f55f3d66d570c8d54f07b WatchSource:0}: Error finding container 11bcb511642b0abcf528ee5d32a703d97ce1e54e897f55f3d66d570c8d54f07b: Status 404 returned error can't find the container with id 11bcb511642b0abcf528ee5d32a703d97ce1e54e897f55f3d66d570c8d54f07b
Sep 17 16:41:52.450410 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.450351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2097712_860e_4e0a_85ef_f699c9d02b3f.slice/crio-abe352963c66f2fb08dd49ae2e635e222be0156281ee7f4ce5ee37316a0848a7 WatchSource:0}: Error finding container abe352963c66f2fb08dd49ae2e635e222be0156281ee7f4ce5ee37316a0848a7: Status 404 returned error can't find the container with id abe352963c66f2fb08dd49ae2e635e222be0156281ee7f4ce5ee37316a0848a7
Sep 17 16:41:52.451192 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.451085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ec5e11_8ebd_4267_bc43_f147545d2b58.slice/crio-6e4555cdb9614688b5f39e6131cf5a5f5887c3dae210800d5730f7e206ff9424 WatchSource:0}: Error finding container 6e4555cdb9614688b5f39e6131cf5a5f5887c3dae210800d5730f7e206ff9424: Status 404 returned error can't find the container with id 6e4555cdb9614688b5f39e6131cf5a5f5887c3dae210800d5730f7e206ff9424
Sep 17 16:41:52.453344 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:41:52.453257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8617572_e20f_4969_ac9d_b6e375107c96.slice/crio-190ae0a439520ce10576530f0aa2962acce989f9e0d44c5f6d6e51464574268b WatchSource:0}: Error finding container 190ae0a439520ce10576530f0aa2962acce989f9e0d44c5f6d6e51464574268b: Status 404 returned error can't find the container with id 190ae0a439520ce10576530f0aa2962acce989f9e0d44c5f6d6e51464574268b
Sep 17 16:41:52.914629 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:52.914340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:52.914777 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.914710    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:52.914842 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:52.914781    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:54.914761936 +0000 UTC m=+7.612264116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.015607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.015667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.015876    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.015897    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.015926    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.015983    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:55.015966288 +0000 UTC m=+7.713468478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.016385    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:53.016471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.016434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:41:55.016418774 +0000 UTC m=+7.713920956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:53.327932 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.326134    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:36:51 +0000 UTC" deadline="2028-04-26 22:32:17.216165258 +0000 UTC"
Sep 17 16:41:53.327932 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.326163    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14093h50m23.890004925s"
Sep 17 16:41:53.358861 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.358237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:53.358861 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.358358    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:41:53.358861 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.358729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:53.358861 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.358815    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:41:53.359266 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.359224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:53.367982 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:53.364042    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:41:53.382141 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.382113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrxq6" event={"ID":"17ec5e11-8ebd-4267-bc43-f147545d2b58","Type":"ContainerStarted","Data":"6e4555cdb9614688b5f39e6131cf5a5f5887c3dae210800d5730f7e206ff9424"}
Sep 17 16:41:53.385393 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.385368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqzcq" event={"ID":"c2097712-860e-4e0a-85ef-f699c9d02b3f","Type":"ContainerStarted","Data":"abe352963c66f2fb08dd49ae2e635e222be0156281ee7f4ce5ee37316a0848a7"}
Sep 17 16:41:53.400226 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.400199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cz9d" event={"ID":"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f","Type":"ContainerStarted","Data":"53095c2195d628c741cc60e503f7acdddd23d9dd365946f780e143d0ac2c4295"}
Sep 17 16:41:53.424928 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.424883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" event={"ID":"607b69b0e690d92464a627a4aea229b6","Type":"ContainerStarted","Data":"d04a5759db29af3854d02c4edffc06a34703c9a287dd7a98e238eb8345b580df"}
Sep 17 16:41:53.435086 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.435060    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c26dd9f0a085e62835ad42e7cd0728a" containerID="9546e3864f1eaf98398cd59b7693fccd88dcfb005410f258cca2ce44e92e4369" exitCode=0
Sep 17 16:41:53.435169 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.435135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"0c26dd9f0a085e62835ad42e7cd0728a","Type":"ContainerDied","Data":"9546e3864f1eaf98398cd59b7693fccd88dcfb005410f258cca2ce44e92e4369"}
Sep 17 16:41:53.454599 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.454574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z24x4" event={"ID":"d6e802b9-240f-4828-b1af-e66cc459d335","Type":"ContainerStarted","Data":"2941c0882d748fafc62c1550793266b8e3a7f7fecbca88969c36a9e97d2d2ea0"}
Sep 17 16:41:53.455810 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.455764    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-112.ec2.internal" podStartSLOduration=3.455749828 podStartE2EDuration="3.455749828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:53.440566025 +0000 UTC m=+6.138068226" watchObservedRunningTime="2026-09-17 16:41:53.455749828 +0000 UTC m=+6.153252027"
Sep 17 16:41:53.459142 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.459120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"190ae0a439520ce10576530f0aa2962acce989f9e0d44c5f6d6e51464574268b"}
Sep 17 16:41:53.462235 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.462212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerStarted","Data":"11bcb511642b0abcf528ee5d32a703d97ce1e54e897f55f3d66d570c8d54f07b"}
Sep 17 16:41:53.470287 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.470223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf" event={"ID":"42053039-27d2-43e0-80a1-22148a863095","Type":"ContainerStarted","Data":"f4ea2c2fa20b886d1f59715f360fcb86ff05d4a0cbf214c623faf863a215b555"}
Sep 17 16:41:53.481370 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.481345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wjl99" event={"ID":"81fac588-8726-472a-8b17-249a1384ddbc","Type":"ContainerStarted","Data":"78d38ed326757ecea34c10936313827e55f2ee1b44d626e13e714aea756208e3"}
Sep 17 16:41:53.489735 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:53.489715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c7z4p" event={"ID":"40ca1de2-f941-4a8f-80b7-b413cf381930","Type":"ContainerStarted","Data":"d62900b23258e9f3238f0022747135c6ab41d4adc1d3c80cf16614c912546311"}
Sep 17 16:41:54.507983 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:54.507948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" event={"ID":"0c26dd9f0a085e62835ad42e7cd0728a","Type":"ContainerStarted","Data":"8809c3a3bf59a58ed37d477f071f1033f01953ceb810ab44c531f33d625c4fc4"}
Sep 17 16:41:54.941708 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:54.941672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:54.941876 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:54.941824    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:54.941969 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:54.941890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:58.941871642 +0000 UTC m=+11.639373836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:55.042265 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:55.042204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:55.042265 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:55.042264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:55.042471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042393    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:55.042471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042412    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:55.042471 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042426    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:55.042623 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042482    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.042465107 +0000 UTC m=+11.739967297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:55.042936 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042899    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:55.043052 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.042969    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:41:59.042955549 +0000 UTC m=+11.740457744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:55.359059 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:55.359021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:55.359246 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:55.359033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:55.359246 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.359154    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:41:55.359246 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:55.359170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:55.359420 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.359277    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:41:55.359420 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:55.359377    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:41:57.359058 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:57.359025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:57.359512 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:57.359168    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:41:57.359512 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:57.359178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:57.359512 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:57.359299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:57.359512 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:57.359413    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:41:57.359512 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:57.359494    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:41:58.976115 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:58.976077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:58.976601 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:58.976215    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:58.976601 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:58.976268    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:06.97625023 +0000 UTC m=+19.673752405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:41:59.076965 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:59.076929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:59.077132 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:59.076992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:59.077206 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077159    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:41:59.077206 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077178    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:41:59.077206 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077190    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:59.077357 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077244    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:07.077226956 +0000 UTC m=+19.774729150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:41:59.077654 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077632    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:59.077741 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.077694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:07.077677404 +0000 UTC m=+19.775179587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:41:59.359725 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:59.359696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:41:59.359949 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.359786    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:41:59.360024 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:59.359996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:41:59.360932 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:41:59.360882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:41:59.361048 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.361003    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:41:59.361110 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:41:59.361088    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:01.358692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:01.358691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:01.358809    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:01.358822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:01.358941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:01.359146 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:01.359048    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:03.357806 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:03.357774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:03.358259 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:03.357780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:03.358259 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:03.357884    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:03.358259 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:03.357969    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:03.358259 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:03.358015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:03.358259 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:03.358113    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:05.357961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:05.357926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:05.357961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:05.357949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:05.358450 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:05.357952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:05.358450 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:05.358052    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:05.358450 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:05.358146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:05.358450 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:05.358206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:07.035539 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.035499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:07.036050 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.035631    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:07.036050 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.035688    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.035675574 +0000 UTC m=+35.733177754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:07.136205 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.136170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:07.136362 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.136267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:07.136362 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136347    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:07.136475 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136363    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:07.136475 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136371    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:07.136475 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136386    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:07.136475 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.136403569 +0000 UTC m=+35.833905759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:07.136475 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.136437    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:23.136429504 +0000 UTC m=+35.833931681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:07.358426 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.358345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:07.358426 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.358386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:07.358644 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:07.358351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:07.358644 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.358476    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:07.358644 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.358586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:07.358644 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:07.358624    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:09.358453 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:09.358422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:09.358867 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:09.358508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:09.358867 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:09.358820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:09.358986 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:09.358885    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:09.358986 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:09.358949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:09.359068 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:09.359007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:10.539922 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.539695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z24x4" event={"ID":"d6e802b9-240f-4828-b1af-e66cc459d335","Type":"ContainerStarted","Data":"cfc4aec2b79b84142d53e1e0ec4933636876e41c31054b4e6be8a010bf9f8690"}
Sep 17 16:42:10.541436 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.541412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"b7f9bfba28494b200d52a0e3265ecf6934e8bf7aa015db92f38c9d8f2d6d4236"}
Sep 17 16:42:10.543126 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.543099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerStarted","Data":"2fbacf2e5140765c1af58d54861cd388c8bddace4f1d04b38bab9a29ec98ac3f"}
Sep 17 16:42:10.544455 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.544428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf" event={"ID":"42053039-27d2-43e0-80a1-22148a863095","Type":"ContainerStarted","Data":"ee0b0646a1f063947318e8f4293199fc26f3fcae308f53133ef88e7341ea2037"}
Sep 17 16:42:10.545982 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.545958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wjl99" event={"ID":"81fac588-8726-472a-8b17-249a1384ddbc","Type":"ContainerStarted","Data":"a6892b69bda3e23b63e3d32124957351d18deaeccb96d474dc8f4357794eb09f"}
Sep 17 16:42:10.547387 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.547368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c7z4p" event={"ID":"40ca1de2-f941-4a8f-80b7-b413cf381930","Type":"ContainerStarted","Data":"b803ef0f7c4bcd8022b612377af2b5a807676ea2c1299df195d990a645beec8c"}
Sep 17 16:42:10.548682 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.548654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqzcq" event={"ID":"c2097712-860e-4e0a-85ef-f699c9d02b3f","Type":"ContainerStarted","Data":"581ed0f89ac5691a08af9141a0a62922b4eff211de403e832fb1090c8cd06b1e"}
Sep 17 16:42:10.549835 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.549818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cz9d" event={"ID":"db51668a-9f53-4ee9-b8c3-1d0a9aaf826f","Type":"ContainerStarted","Data":"b7835178a4e63083841089514b8555002d152fd7a0a1c27faf5f7b03f6920acf"}
Sep 17 16:42:10.557689 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.557655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-112.ec2.internal" podStartSLOduration=20.557644132 podStartE2EDuration="20.557644132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:54.528550471 +0000 UTC m=+7.226052669" watchObservedRunningTime="2026-09-17 16:42:10.557644132 +0000 UTC m=+23.255146330"
Sep 17 16:42:10.557937 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.557897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z24x4" podStartSLOduration=3.869189232 podStartE2EDuration="21.557891992s" totalImagesPullingTime="17.68870276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.557595984 +0000 UTC m=+23.255098181" watchObservedRunningTime="2026-09-17 16:42:10.557891992 +0000 UTC m=+23.255394189"
Sep 17 16:42:10.601521 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.601252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xqzcq" podStartSLOduration=4.342064404 podStartE2EDuration="21.601231292s" totalImagesPullingTime="17.259166888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.576573596 +0000 UTC m=+23.274075794" watchObservedRunningTime="2026-09-17 16:42:10.601231292 +0000 UTC m=+23.298733490"
Sep 17 16:42:10.624466 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.624412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c7z4p" podStartSLOduration=3.823422396 podStartE2EDuration="21.624395857s" totalImagesPullingTime="17.800973461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.62413405 +0000 UTC m=+23.321636249" watchObservedRunningTime="2026-09-17 16:42:10.624395857 +0000 UTC m=+23.321898055"
Sep 17 16:42:10.655978 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.655936    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wjl99" podStartSLOduration=3.919940759 podStartE2EDuration="21.65590152s" totalImagesPullingTime="17.735960761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.654083481 +0000 UTC m=+23.351585679" watchObservedRunningTime="2026-09-17 16:42:10.65590152 +0000 UTC m=+23.353403718"
Sep 17 16:42:10.681064 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:10.681022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4cz9d" podStartSLOduration=3.943581546 podStartE2EDuration="21.681005731s" totalImagesPullingTime="17.737424185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:10.680866986 +0000 UTC m=+23.378369212" watchObservedRunningTime="2026-09-17 16:42:10.681005731 +0000 UTC m=+23.378507929"
Sep 17 16:42:11.358295 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.358117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:11.358459 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.358118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:11.358459 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:11.358399    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:11.358459 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.358117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:11.358642 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:11.358457    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:11.358642 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:11.358514    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:11.554304 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554277    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554585    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8617572-e20f-4969-ac9d-b6e375107c96" containerID="8bf85adbca4560531a7f79be114aee0d03c41b69b0c56a9c2225a5c19d921036" exitCode=1
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"5f01252796a82d5dca66997a74fd1ebdb00cafd261fc344ce358082a9ad35155"}
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"719bfac2a02342e5245d6ec2670f75fdae2622cc55b36855f1fec58271002cf0"}
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"19e506c158e6bf3b9fcaded7ee66cbb5df909f8dd8743b77f7dd586b7f297433"}
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"36da97746ca2f5e6125681b547ef7e3a4c236e5aa44d119edcc6383d248958e3"}
Sep 17 16:42:11.554961 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.554886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerDied","Data":"8bf85adbca4560531a7f79be114aee0d03c41b69b0c56a9c2225a5c19d921036"}
Sep 17 16:42:11.556028 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.556006    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="2fbacf2e5140765c1af58d54861cd388c8bddace4f1d04b38bab9a29ec98ac3f" exitCode=0
Sep 17 16:42:11.556118 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.556096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"2fbacf2e5140765c1af58d54861cd388c8bddace4f1d04b38bab9a29ec98ac3f"}
Sep 17 16:42:11.730191 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:11.730166    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:42:12.365521 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:12.365413    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:42:11.730183781Z","UUID":"7dd0a308-5483-4c7f-b01d-d21464ec8718","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:42:12.366978 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:12.366959    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:42:12.367103 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:12.366989    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:42:12.561184 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:12.561141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf" event={"ID":"42053039-27d2-43e0-80a1-22148a863095","Type":"ContainerStarted","Data":"6534dc60e9a8d8555efba0c1a377dd573a62f56553ebb47f44d462ea2e8f3999"}
Sep 17 16:42:12.563097 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:12.563069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrxq6" event={"ID":"17ec5e11-8ebd-4267-bc43-f147545d2b58","Type":"ContainerStarted","Data":"95748ee001704243f48b5002c45b2b36c60f677c74a71b8919cc9cd9d9e9358d"}
Sep 17 16:42:13.358114 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.358080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:13.358307 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.358080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:13.358307 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:13.358198    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:13.358307 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.358255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:13.358472 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:13.358333    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:13.358519 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:13.358462    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:13.569306 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.569281    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:42:13.569727 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.569700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"7bf31a088d794354386757363c2af8006a58c41598de5970977a18dd4e18fc16"}
Sep 17 16:42:13.572082 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.572055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf" event={"ID":"42053039-27d2-43e0-80a1-22148a863095","Type":"ContainerStarted","Data":"5b53efbe3aec6b7dbec07f2a5c6a6458981078eac5fff1676fb6e98eb2ec71d9"}
Sep 17 16:42:13.592768 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.592717    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdwnf" podStartSLOduration=4.765146285 podStartE2EDuration="24.592661017s" totalImagesPullingTime="19.827514732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:13.592326323 +0000 UTC m=+26.289828521" watchObservedRunningTime="2026-09-17 16:42:13.592661017 +0000 UTC m=+26.290163216"
Sep 17 16:42:13.593069 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:13.593040    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wrxq6" podStartSLOduration=6.867066682 podStartE2EDuration="24.593031609s" totalImagesPullingTime="17.725964927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:12.585724878 +0000 UTC m=+25.283227077" watchObservedRunningTime="2026-09-17 16:42:13.593031609 +0000 UTC m=+26.290533810"
Sep 17 16:42:15.250920 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.250872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:42:15.251500 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.251478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:42:15.358549 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.358473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:15.358707 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.358479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:15.358707 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:15.358597    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:15.358707 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.358484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:15.358707 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:15.358667    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:15.358889 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:15.358766    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:15.576599 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.576476    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:42:15.576880 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:15.576865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xqzcq"
Sep 17 16:42:16.581514 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.581487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:42:16.582301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.581924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"574619f5e34b78506b2f3a6d4e279569c1e4978afd5ceb0c14c67c516d41a773"}
Sep 17 16:42:16.582301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.582170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:16.582420 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.582372    2580 scope.go:117] "RemoveContainer" containerID="8bf85adbca4560531a7f79be114aee0d03c41b69b0c56a9c2225a5c19d921036"
Sep 17 16:42:16.585818 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.585791    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="4606d89dc854956543892bc9daf457f0e489b4b8eaf94a4f1b0078afe5cc8b5a" exitCode=0
Sep 17 16:42:16.585956 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.585874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"4606d89dc854956543892bc9daf457f0e489b4b8eaf94a4f1b0078afe5cc8b5a"}
Sep 17 16:42:16.599999 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:16.599975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:17.358283 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.358103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:17.358474 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.358106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:17.358474 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.358391    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:17.358474 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.358115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:17.358671 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.358438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:17.358671 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.358550    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:17.541949 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.541920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k8mwq"]
Sep 17 16:42:17.544674 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.544597    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzlws"]
Sep 17 16:42:17.545121 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.545100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jznpc"]
Sep 17 16:42:17.592688 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.592670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:42:17.593108 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.593086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" event={"ID":"f8617572-e20f-4969-ac9d-b6e375107c96","Type":"ContainerStarted","Data":"f65af09aa2fbe5ddd69e9f650920eefbd89c3221ba0e610c71dfec1e4131c8db"}
Sep 17 16:42:17.593260 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.593243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:17.593358 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.593304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:17.595727 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.595369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerStarted","Data":"f2343948b5424669f52a3e9778909e4d6a0cd07ce76ba63075ada78f503fdcfd"}
Sep 17 16:42:17.595727 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.595387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:17.595727 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.595486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:17.595727 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.595489    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:17.595727 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.595683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:17.596021 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.595790    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:17.596021 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:17.595901    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:17.608829 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.608811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:17.626857 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:17.626810    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf" podStartSLOduration=10.880259427 podStartE2EDuration="28.626797716s" totalImagesPullingTime="17.746538289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:17.625644237 +0000 UTC m=+30.323146436" watchObservedRunningTime="2026-09-17 16:42:17.626797716 +0000 UTC m=+30.324299914"
Sep 17 16:42:18.602926 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:18.602878    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="f2343948b5424669f52a3e9778909e4d6a0cd07ce76ba63075ada78f503fdcfd" exitCode=0
Sep 17 16:42:18.603361 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:18.602962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"f2343948b5424669f52a3e9778909e4d6a0cd07ce76ba63075ada78f503fdcfd"}
Sep 17 16:42:19.361355 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:19.361332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:19.361502 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:19.361440    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:19.361502 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:19.361332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:19.361502 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:19.361332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:19.361675 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:19.361545    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:19.361675 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:19.361618    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:20.609283 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:20.609242    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="2b18e75e4761e5ae1fb0c1923a7f7efe5d0aceb6d25cb2a8be0fa36280e6610d" exitCode=0
Sep 17 16:42:20.609673 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:20.609287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"2b18e75e4761e5ae1fb0c1923a7f7efe5d0aceb6d25cb2a8be0fa36280e6610d"}
Sep 17 16:42:21.358064 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:21.358036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:21.358224 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:21.358099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:21.358224 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:21.358118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:21.358337 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:21.358217    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:21.358528 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:21.358497    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:21.358629 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:21.358582    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:23.049200 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.049163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:23.049668 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.049311    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:23.049668 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.049380    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret podName:2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:55.049363808 +0000 UTC m=+67.746865984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret") pod "global-pull-secret-syncer-rzlws" (UID: "2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:23.149560 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.149487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:23.149560 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.149529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:23.149759 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149658    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:23.149759 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149683    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:23.149759 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149710    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:23.149759 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149723    2580 projected.go:196] Error preparing data for projected volume kube-api-access-rqbgm for pod openshift-network-diagnostics/network-check-target-jznpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:23.149759 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149740    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:42:55.149718795 +0000 UTC m=+67.847220992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:23.149956 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.149770    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm podName:116f607f-61d0-4a92-9045-d7e5b6cba679 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:55.149757952 +0000 UTC m=+67.847260128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rqbgm" (UniqueName: "kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm") pod "network-check-target-jznpc" (UID: "116f607f-61d0-4a92-9045-d7e5b6cba679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:23.358104 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.358070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:23.358104 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.358109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:23.358398 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.358070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:23.358398 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.358202    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzlws" podUID="2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43"
Sep 17 16:42:23.358398 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.358310    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k8mwq" podUID="9336b6a4-be76-4751-92fa-9372275f0ddd"
Sep 17 16:42:23.358398 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.358384    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jznpc" podUID="116f607f-61d0-4a92-9045-d7e5b6cba679"
Sep 17 16:42:23.365695 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.365667    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-112.ec2.internal" event="NodeReady"
Sep 17 16:42:23.365839 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.365800    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:42:23.417733 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.417656    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:42:23.439845 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.439819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q49bh"]
Sep 17 16:42:23.440015 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.439985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.443790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.443757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 16:42:23.443790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.443777    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 16:42:23.444003 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.443972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbjg6\""
Sep 17 16:42:23.444096 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.444048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 16:42:23.456492 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.456464    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w5hds"]
Sep 17 16:42:23.456645 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.456623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.459778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.459746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:42:23.461414 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.460766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bdrbl\""
Sep 17 16:42:23.461414 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.460900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:42:23.463011 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.462993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 16:42:23.474395 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.474374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:42:23.474476 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.474402    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q49bh"]
Sep 17 16:42:23.474476 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.474417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5hds"]
Sep 17 16:42:23.474568 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.474498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.476825 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.476808    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:42:23.476980 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.476960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rc4fv\""
Sep 17 16:42:23.477096 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.477078    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:42:23.477220 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.477209    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:42:23.558835 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.558807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095872e5-4dd2-4526-abec-91cea5f04f2d-tmp-dir\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.558845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/095872e5-4dd2-4526-abec-91cea5f04f2d-config-volume\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.558883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.558926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.558973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nhw\" (UniqueName: \"kubernetes.io/projected/095872e5-4dd2-4526-abec-91cea5f04f2d-kube-api-access-n5nhw\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r77c\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.559319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.559319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.559269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660586 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-cert\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nhw\" (UniqueName: \"kubernetes.io/projected/095872e5-4dd2-4526-abec-91cea5f04f2d-kube-api-access-n5nhw\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r77c\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.660803 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929tw\" (UniqueName: \"kubernetes.io/projected/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-kube-api-access-929tw\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.660952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.661040    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.661052    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.661070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.661103    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:24.161091291 +0000 UTC m=+36.858593467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:23.661197 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.661192    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:23.661644 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:23.661251    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:24.16123423 +0000 UTC m=+36.858736421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:23.661644 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.661293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095872e5-4dd2-4526-abec-91cea5f04f2d-tmp-dir\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.661644 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.661347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/095872e5-4dd2-4526-abec-91cea5f04f2d-config-volume\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.661816 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.661646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095872e5-4dd2-4526-abec-91cea5f04f2d-tmp-dir\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.661946 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.661927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/095872e5-4dd2-4526-abec-91cea5f04f2d-config-volume\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.662084 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.662062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.662319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.662294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.666559 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.666541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.666647 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.666540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.672394 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.672337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r77c\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.673262 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.673239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:23.673356 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.673275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nhw\" (UniqueName: \"kubernetes.io/projected/095872e5-4dd2-4526-abec-91cea5f04f2d-kube-api-access-n5nhw\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:23.762336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.762306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-cert\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.762525 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.762386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929tw\" (UniqueName: \"kubernetes.io/projected/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-kube-api-access-929tw\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.765290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.765266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-cert\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.773509 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.773488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-929tw\" (UniqueName: \"kubernetes.io/projected/32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7-kube-api-access-929tw\") pod \"ingress-canary-w5hds\" (UID: \"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7\") " pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:23.784322 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:23.784300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5hds"
Sep 17 16:42:24.166312 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:24.166276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:24.166362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:24.166450    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:24.166472    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:24.166524    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:24.166559    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.166539321 +0000 UTC m=+37.864041512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:24.167080 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:24.166583    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:25.166566093 +0000 UTC m=+37.864068281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:25.132398 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.132267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5hds"]
Sep 17 16:42:25.136212 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:42:25.136187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b1767d_f4fe_4c8c_8cc6_c23f21dc5fe7.slice/crio-ff5f5f2cfda05320044d5c392224ed9c2569bf800cb9ffc046ac41183ae7e30a WatchSource:0}: Error finding container ff5f5f2cfda05320044d5c392224ed9c2569bf800cb9ffc046ac41183ae7e30a: Status 404 returned error can't find the container with id ff5f5f2cfda05320044d5c392224ed9c2569bf800cb9ffc046ac41183ae7e30a
Sep 17 16:42:25.174115 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.174089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.174167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:25.174272    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:25.174357    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:27.174333999 +0000 UTC m=+39.871836176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:25.174277    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:25.174392    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:25.174664 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:25.174447    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:27.174430802 +0000 UTC m=+39.871933010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:25.358114 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.358029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:25.358114 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.358059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:25.358232 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.358030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:25.360896 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.360794    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:25.360896 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.360848    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:25.360896 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.360862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:25.361125 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.360898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:25.361735 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.361717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n5btm\""
Sep 17 16:42:25.361817 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.361732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dfgr8\""
Sep 17 16:42:25.624749 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.624660    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="f4dbe5edc61f6f6fa7a8d41b0c6f588c6a7a61053e3a404731852c93fd15a024" exitCode=0
Sep 17 16:42:25.624942 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.624740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"f4dbe5edc61f6f6fa7a8d41b0c6f588c6a7a61053e3a404731852c93fd15a024"}
Sep 17 16:42:25.625841 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:25.625823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5hds" event={"ID":"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7","Type":"ContainerStarted","Data":"ff5f5f2cfda05320044d5c392224ed9c2569bf800cb9ffc046ac41183ae7e30a"}
Sep 17 16:42:26.631852 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:26.631816    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc823426-4ef1-4d6e-86be-aff70a072c07" containerID="f655bd96409da83d370627459dd4fdaa315be95a0fd268977fc9cb43ae3485b4" exitCode=0
Sep 17 16:42:26.632228 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:26.631854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerDied","Data":"f655bd96409da83d370627459dd4fdaa315be95a0fd268977fc9cb43ae3485b4"}
Sep 17 16:42:27.189047 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.189006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:27.189215 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.189117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:27.189215 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:27.189175    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:27.189215 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:27.189199    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:27.189357 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:27.189237    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:27.189357 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:27.189259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:31.189242788 +0000 UTC m=+43.886744969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:27.189357 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:27.189287    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:31.189271048 +0000 UTC m=+43.886773230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:27.638543 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.638465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" event={"ID":"fc823426-4ef1-4d6e-86be-aff70a072c07","Type":"ContainerStarted","Data":"74ae4d58e10d7c0f5764d09858e22aa67270c2d2137a122164d1ab693b4d73fe"}
Sep 17 16:42:27.639950 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.639926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5hds" event={"ID":"32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7","Type":"ContainerStarted","Data":"868a06578572e9a21635bfeaf101fe2965a7bbf64d696e22913c9d147a864807"}
Sep 17 16:42:27.662668 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.662626    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kx7rd" podStartSLOduration=10.49642714 podStartE2EDuration="38.662612244s" totalImagesPullingTime="28.166185104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:27.661672884 +0000 UTC m=+40.359175081" watchObservedRunningTime="2026-09-17 16:42:27.662612244 +0000 UTC m=+40.360114442"
Sep 17 16:42:27.684061 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:27.684024    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w5hds" podStartSLOduration=2.426000563 podStartE2EDuration="4.684014642s" totalImagesPullingTime="2.258014079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:27.68263134 +0000 UTC m=+40.380133537" watchObservedRunningTime="2026-09-17 16:42:27.684014642 +0000 UTC m=+40.381516839"
Sep 17 16:42:31.217167 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:31.217126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:31.217185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:31.217260    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:31.217278    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:31.217289    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:31.217329    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.217311576 +0000 UTC m=+51.914813752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:31.217603 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:31.217343    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.217336772 +0000 UTC m=+51.914838948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:39.269837 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:39.269799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:39.269876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:39.269963    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:39.270024    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:39.270041    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:39.270026    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:55.27001152 +0000 UTC m=+67.967513696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:39.270252 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:39.270107    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:55.270090593 +0000 UTC m=+67.967592785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:49.616264 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:49.616234    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4plf"
Sep 17 16:42:55.076358 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.076322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:55.078636 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.078616    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:55.090693 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.090666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43-original-pull-secret\") pod \"global-pull-secret-syncer-rzlws\" (UID: \"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43\") " pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:55.177004 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.176975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:55.177121 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.177046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:42:55.179150 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.179129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:55.179239 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.179223    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:55.187703 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.187683    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 16:42:55.187782 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.187750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs podName:9336b6a4-be76-4751-92fa-9372275f0ddd nodeName:}" failed. No retries permitted until 2026-09-17 16:43:59.187729797 +0000 UTC m=+131.885231973 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs") pod "network-metrics-daemon-k8mwq" (UID: "9336b6a4-be76-4751-92fa-9372275f0ddd") : secret "metrics-daemon-secret" not found
Sep 17 16:42:55.189927 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.189898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:55.200197 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.200181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbgm\" (UniqueName: \"kubernetes.io/projected/116f607f-61d0-4a92-9045-d7e5b6cba679-kube-api-access-rqbgm\") pod \"network-check-target-jznpc\" (UID: \"116f607f-61d0-4a92-9045-d7e5b6cba679\") " pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:55.277913 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.277884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:42:55.278040 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.277966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:42:55.278107 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.278055    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:55.278107 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.278054    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 16:42:55.278107 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.278080    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f6665948-pnfcj: secret "image-registry-tls" not found
Sep 17 16:42:55.278107 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.278106    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls podName:095872e5-4dd2-4526-abec-91cea5f04f2d nodeName:}" failed. No retries permitted until 2026-09-17 16:43:27.278090918 +0000 UTC m=+99.975593099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls") pod "dns-default-q49bh" (UID: "095872e5-4dd2-4526-abec-91cea5f04f2d") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:55.278259 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:42:55.278146    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls podName:4cbb41c4-0eed-4850-99aa-f2ac252ad518 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:27.278127639 +0000 UTC m=+99.975629835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls") pod "image-registry-69f6665948-pnfcj" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518") : secret "image-registry-tls" not found
Sep 17 16:42:55.367738 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.367675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzlws"
Sep 17 16:42:55.380054 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.380032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dfgr8\""
Sep 17 16:42:55.388751 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.388733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:42:55.588166 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.588135    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sd2q8"]
Sep 17 16:42:55.590072 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.590057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.593586 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.593563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7rkvp\""
Sep 17 16:42:55.593934 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.593590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 16:42:55.594116 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.594094    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:42:55.594232 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.594188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:42:55.594393 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.594375    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 16:42:55.598415 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.598397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 16:42:55.602022 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.602002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sd2q8"]
Sep 17 16:42:55.639575 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.639512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzlws"]
Sep 17 16:42:55.643550 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:42:55.643526    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b13d7a0_ed59_45f9_8ec6_6c9f9c69ad43.slice/crio-724bae9625d92a8b091aa257a99fa93f297a83b4676e9193c3f0bb2d7f536767 WatchSource:0}: Error finding container 724bae9625d92a8b091aa257a99fa93f297a83b4676e9193c3f0bb2d7f536767: Status 404 returned error can't find the container with id 724bae9625d92a8b091aa257a99fa93f297a83b4676e9193c3f0bb2d7f536767
Sep 17 16:42:55.659256 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.659237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jznpc"]
Sep 17 16:42:55.662268 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:42:55.662246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116f607f_61d0_4a92_9045_d7e5b6cba679.slice/crio-1e35e157c03865d7cf5e86d3b1afd3aca4e3b716bd3bba0b3d731610d78451cd WatchSource:0}: Error finding container 1e35e157c03865d7cf5e86d3b1afd3aca4e3b716bd3bba0b3d731610d78451cd: Status 404 returned error can't find the container with id 1e35e157c03865d7cf5e86d3b1afd3aca4e3b716bd3bba0b3d731610d78451cd
Sep 17 16:42:55.680635 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-tmp\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.680719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-snapshots\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.680719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.680719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58llj\" (UniqueName: \"kubernetes.io/projected/6073644f-97cc-4086-a8da-3b606a68d09b-kube-api-access-58llj\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.680831 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.680831 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.680797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6073644f-97cc-4086-a8da-3b606a68d09b-serving-cert\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.708557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.708530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jznpc" event={"ID":"116f607f-61d0-4a92-9045-d7e5b6cba679","Type":"ContainerStarted","Data":"1e35e157c03865d7cf5e86d3b1afd3aca4e3b716bd3bba0b3d731610d78451cd"}
Sep 17 16:42:55.709833 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.709813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzlws" event={"ID":"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43","Type":"ContainerStarted","Data":"724bae9625d92a8b091aa257a99fa93f297a83b4676e9193c3f0bb2d7f536767"}
Sep 17 16:42:55.781550 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6073644f-97cc-4086-a8da-3b606a68d09b-serving-cert\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781657 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-tmp\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781712 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-snapshots\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781753 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781801 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58llj\" (UniqueName: \"kubernetes.io/projected/6073644f-97cc-4086-a8da-3b606a68d09b-kube-api-access-58llj\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781868 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.781959 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.781937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-tmp\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.782229 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.782208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6073644f-97cc-4086-a8da-3b606a68d09b-snapshots\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.782380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.782363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.782766 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.782749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6073644f-97cc-4086-a8da-3b606a68d09b-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.784042 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.784025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6073644f-97cc-4086-a8da-3b606a68d09b-serving-cert\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.795399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.795377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58llj\" (UniqueName: \"kubernetes.io/projected/6073644f-97cc-4086-a8da-3b606a68d09b-kube-api-access-58llj\") pod \"insights-operator-6f9c644dd9-sd2q8\" (UID: \"6073644f-97cc-4086-a8da-3b606a68d09b\") " pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:55.903325 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:55.903261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8"
Sep 17 16:42:56.173033 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:56.172951    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sd2q8"]
Sep 17 16:42:56.176418 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:42:56.176384    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6073644f_97cc_4086_a8da_3b606a68d09b.slice/crio-ee4ae90498fd8b5f531f92138e81ea76e0629b393cb4a9c26a5b3dfa95c9eed6 WatchSource:0}: Error finding container ee4ae90498fd8b5f531f92138e81ea76e0629b393cb4a9c26a5b3dfa95c9eed6: Status 404 returned error can't find the container with id ee4ae90498fd8b5f531f92138e81ea76e0629b393cb4a9c26a5b3dfa95c9eed6
Sep 17 16:42:56.714339 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:42:56.714301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8" event={"ID":"6073644f-97cc-4086-a8da-3b606a68d09b","Type":"ContainerStarted","Data":"ee4ae90498fd8b5f531f92138e81ea76e0629b393cb4a9c26a5b3dfa95c9eed6"}
Sep 17 16:43:00.727374 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:00.727319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jznpc" event={"ID":"116f607f-61d0-4a92-9045-d7e5b6cba679","Type":"ContainerStarted","Data":"3dd0206c50f622e50b33b55cbb084ea576c4517328811c96ac4216f0b2b27a1f"}
Sep 17 16:43:00.727745 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:00.727449    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:43:00.729207 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:00.729183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8" event={"ID":"6073644f-97cc-4086-a8da-3b606a68d09b","Type":"ContainerStarted","Data":"4e4feb649dc5486ee59530f814aae69768c483fdb3a8589ecb1edff06e33ce6c"}
Sep 17 16:43:00.744266 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:00.744109    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jznpc" podStartSLOduration=66.768189789 podStartE2EDuration="1m11.744097192s" totalImagesPullingTime="4.975907403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:00.743416177 +0000 UTC m=+73.440918375" watchObservedRunningTime="2026-09-17 16:43:00.744097192 +0000 UTC m=+73.441599391"
Sep 17 16:43:00.764872 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:00.764083    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8" podStartSLOduration=1.299453006 podStartE2EDuration="5.764068032s" totalImagesPullingTime="4.464615026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:00.763771888 +0000 UTC m=+73.461274087" watchObservedRunningTime="2026-09-17 16:43:00.764068032 +0000 UTC m=+73.461570230"
Sep 17 16:43:01.733220 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.733185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzlws" event={"ID":"2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43","Type":"ContainerStarted","Data":"eb39de1568d2f74cc4fa4b49fbf4a4e329628cfe55d62851f9bd1b384c4fdff9"}
Sep 17 16:43:01.760430 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.760377    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rzlws" podStartSLOduration=67.750228013 podStartE2EDuration="1m12.760359922s" totalImagesPullingTime="5.010131909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:01.759534939 +0000 UTC m=+74.457037136" watchObservedRunningTime="2026-09-17 16:43:01.760359922 +0000 UTC m=+74.457862120"
Sep 17 16:43:01.897954 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.897896    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dmv8b"]
Sep 17 16:43:01.901050 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.901034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:01.904106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.904086    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:43:01.904330 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.904319    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jqstw\""
Sep 17 16:43:01.904385 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.904357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:43:01.913308 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.913286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmv8b"]
Sep 17 16:43:01.931398 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.931370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k9t\" (UniqueName: \"kubernetes.io/projected/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-api-access-54k9t\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:01.931490 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.931412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-crio-socket\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:01.931545 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.931523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:01.931585 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.931546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-data-volume\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:01.934658 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:01.931573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032817 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-crio-socket\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032891 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032891 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-data-volume\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032992 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032992 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k9t\" (UniqueName: \"kubernetes.io/projected/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-api-access-54k9t\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.032992 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.032942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-crio-socket\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.033090 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.033004    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:43:02.033090 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.033045    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:02.033090 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.033061    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:02.533046656 +0000 UTC m=+75.230548832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : configmap "kube-rbac-proxy" not found
Sep 17 16:43:02.033198 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.033094    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:02.533081207 +0000 UTC m=+75.230583383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:02.033276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.033261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-data-volume\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.041809 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.041792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k9t\" (UniqueName: \"kubernetes.io/projected/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-api-access-54k9t\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.537721 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.537690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.537860 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:02.537734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:02.537860 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.537802    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:43:02.537860 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.537833    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:02.537976 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.537862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:03.537847139 +0000 UTC m=+76.235349316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : configmap "kube-rbac-proxy" not found
Sep 17 16:43:02.537976 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:02.537880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:03.537869586 +0000 UTC m=+76.235371762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:03.430664 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:03.430641    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z24x4_d6e802b9-240f-4828-b1af-e66cc459d335/dns-node-resolver/0.log"
Sep 17 16:43:03.545319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:03.545272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:03.545319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:03.545321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:03.545523 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:03.545391    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:43:03.545523 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:03.545439    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:03.545523 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:03.545458    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:05.545443185 +0000 UTC m=+78.242945361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : configmap "kube-rbac-proxy" not found
Sep 17 16:43:03.545523 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:03.545488    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:05.545476726 +0000 UTC m=+78.242978926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:04.030969 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:04.030936    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4cz9d_db51668a-9f53-4ee9-b8c3-1d0a9aaf826f/node-ca/0.log"
Sep 17 16:43:04.631832 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:04.631796    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5hds_32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7/serve-healthcheck-canary/0.log"
Sep 17 16:43:05.554693 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.554647    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"]
Sep 17 16:43:05.557879 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.557858    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"]
Sep 17 16:43:05.558014 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.557998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"
Sep 17 16:43:05.558095 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.558075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:05.558139 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.558111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:05.558238 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.558222    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:05.558368 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.558250    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:43:05.558368 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.558279    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:09.558263398 +0000 UTC m=+82.255765589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:05.558368 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.558340    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:09.558320423 +0000 UTC m=+82.255822615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : configmap "kube-rbac-proxy" not found
Sep 17 16:43:05.560425 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.560404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:43:05.560626 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.560613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.560673 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.560635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-v85xd\""
Sep 17 16:43:05.561168 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.561148    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 16:43:05.568339 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.568319    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 16:43:05.568339 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.568330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:43:05.568486 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.568325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 16:43:05.568486 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.568410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ql2t6\""
Sep 17 16:43:05.568486 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.568365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 16:43:05.591797 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.591772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"]
Sep 17 16:43:05.593011 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.592991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"]
Sep 17 16:43:05.659086 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.659058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jcq\" (UniqueName: \"kubernetes.io/projected/5eef0184-be01-43ad-a56d-3f5819f04916-kube-api-access-h7jcq\") pod \"volume-data-source-validator-5c4db8bdf9-lvfdc\" (UID: \"5eef0184-be01-43ad-a56d-3f5819f04916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"
Sep 17 16:43:05.659380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.659113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b50784ed-9d9d-440c-8075-b24d6b16a4fb-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.659380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.659136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr52r\" (UniqueName: \"kubernetes.io/projected/b50784ed-9d9d-440c-8075-b24d6b16a4fb-kube-api-access-dr52r\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.659380 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.659152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b50784ed-9d9d-440c-8075-b24d6b16a4fb-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.702869 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.702845    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"]
Sep 17 16:43:05.705928 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.705894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.712351 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.712324    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-samples-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 16:43:05.712457 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.712324    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-samples-operator-dockercfg-q4wzs\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-cluster-samples-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q4wzs\"" type="*v1.Secret"
Sep 17 16:43:05.722420 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.722400    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-f9cwr"]
Sep 17 16:43:05.724347 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.724299    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-samples-operator-685d7d68b5-wv29x\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-cluster-samples-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" podUID="72346a52-84ef-4303-b475-17b6ea1f15d2" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.724551 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.724514    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"samples-operator-tls\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-cluster-samples-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret"
Sep 17 16:43:05.724745 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:05.724690    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-112.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-samples-operator\": no relationship found between node 'ip-10-0-143-112.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 16:43:05.728508 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.728489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.730393 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.730368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"]
Sep 17 16:43:05.730863 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.730847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 16:43:05.741384 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.741363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 16:43:05.741636 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.741619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:43:05.741718 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.741660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g7ht9\""
Sep 17 16:43:05.741718 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.741663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 16:43:05.742252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.742235    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 16:43:05.759652 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-serving-cert\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.759724 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmvw\" (UniqueName: \"kubernetes.io/projected/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-kube-api-access-rhmvw\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.759724 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jcq\" (UniqueName: \"kubernetes.io/projected/5eef0184-be01-43ad-a56d-3f5819f04916-kube-api-access-h7jcq\") pod \"volume-data-source-validator-5c4db8bdf9-lvfdc\" (UID: \"5eef0184-be01-43ad-a56d-3f5819f04916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"
Sep 17 16:43:05.759878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b50784ed-9d9d-440c-8075-b24d6b16a4fb-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.759878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr52r\" (UniqueName: \"kubernetes.io/projected/b50784ed-9d9d-440c-8075-b24d6b16a4fb-kube-api-access-dr52r\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.759878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-trusted-ca\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.759878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b50784ed-9d9d-440c-8075-b24d6b16a4fb-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.760126 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.759934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-config\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.760126 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.760017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkf5\" (UniqueName: \"kubernetes.io/projected/72346a52-84ef-4303-b475-17b6ea1f15d2-kube-api-access-qjkf5\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.760126 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.760038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.760369 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.760351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b50784ed-9d9d-440c-8075-b24d6b16a4fb-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.761302 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.761288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-f9cwr"]
Sep 17 16:43:05.762104 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.762087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b50784ed-9d9d-440c-8075-b24d6b16a4fb-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.803098 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.803074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr52r\" (UniqueName: \"kubernetes.io/projected/b50784ed-9d9d-440c-8075-b24d6b16a4fb-kube-api-access-dr52r\") pod \"kube-storage-version-migrator-operator-55889f7b47-mskw4\" (UID: \"b50784ed-9d9d-440c-8075-b24d6b16a4fb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.803578 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.803561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jcq\" (UniqueName: \"kubernetes.io/projected/5eef0184-be01-43ad-a56d-3f5819f04916-kube-api-access-h7jcq\") pod \"volume-data-source-validator-5c4db8bdf9-lvfdc\" (UID: \"5eef0184-be01-43ad-a56d-3f5819f04916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"
Sep 17 16:43:05.861163 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkf5\" (UniqueName: \"kubernetes.io/projected/72346a52-84ef-4303-b475-17b6ea1f15d2-kube-api-access-qjkf5\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.861163 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:05.861312 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-serving-cert\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.861312 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmvw\" (UniqueName: \"kubernetes.io/projected/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-kube-api-access-rhmvw\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.861394 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-trusted-ca\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.861394 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.861374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-config\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.862042 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.862021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-config\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.862515 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.862492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-trusted-ca\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.863425 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.863405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-serving-cert\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:05.867376 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.867360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"
Sep 17 16:43:05.872704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.872685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"
Sep 17 16:43:05.873374 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:05.873352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmvw\" (UniqueName: \"kubernetes.io/projected/0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5-kube-api-access-rhmvw\") pod \"console-operator-fd4bb46c-f9cwr\" (UID: \"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5\") " pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:06.037031 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.037000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:06.148882 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.148824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4"]
Sep 17 16:43:06.151528 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.151505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc"]
Sep 17 16:43:06.152254 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:06.152220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50784ed_9d9d_440c_8075_b24d6b16a4fb.slice/crio-af3eb8d8d0c11b3da46b0eb7ade65e85098a916ac158ea333308a91952f3e4f8 WatchSource:0}: Error finding container af3eb8d8d0c11b3da46b0eb7ade65e85098a916ac158ea333308a91952f3e4f8: Status 404 returned error can't find the container with id af3eb8d8d0c11b3da46b0eb7ade65e85098a916ac158ea333308a91952f3e4f8
Sep 17 16:43:06.156858 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:06.156836    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eef0184_be01_43ad_a56d_3f5819f04916.slice/crio-94c2843254ea0a6ab305ec1db698781ab26f3cb479b49bc500aa5583b9a2c78c WatchSource:0}: Error finding container 94c2843254ea0a6ab305ec1db698781ab26f3cb479b49bc500aa5583b9a2c78c: Status 404 returned error can't find the container with id 94c2843254ea0a6ab305ec1db698781ab26f3cb479b49bc500aa5583b9a2c78c
Sep 17 16:43:06.313628 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.313607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-f9cwr"]
Sep 17 16:43:06.315691 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:06.315672    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff77b6f_6fb4_4b00_ac5a_dc4148714ac5.slice/crio-b44e1e821003ab23214284d4293e34f30b8c66de04307d7622b2905730ccf6b5 WatchSource:0}: Error finding container b44e1e821003ab23214284d4293e34f30b8c66de04307d7622b2905730ccf6b5: Status 404 returned error can't find the container with id b44e1e821003ab23214284d4293e34f30b8c66de04307d7622b2905730ccf6b5
Sep 17 16:43:06.723492 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.723457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q4wzs\""
Sep 17 16:43:06.749629 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.749594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc" event={"ID":"5eef0184-be01-43ad-a56d-3f5819f04916","Type":"ContainerStarted","Data":"94c2843254ea0a6ab305ec1db698781ab26f3cb479b49bc500aa5583b9a2c78c"}
Sep 17 16:43:06.751078 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.751017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" event={"ID":"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5","Type":"ContainerStarted","Data":"b44e1e821003ab23214284d4293e34f30b8c66de04307d7622b2905730ccf6b5"}
Sep 17 16:43:06.752364 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.752322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4" event={"ID":"b50784ed-9d9d-440c-8075-b24d6b16a4fb","Type":"ContainerStarted","Data":"af3eb8d8d0c11b3da46b0eb7ade65e85098a916ac158ea333308a91952f3e4f8"}
Sep 17 16:43:06.752642 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.752625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 16:43:06.862014 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:06.861975    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: failed to sync secret cache: timed out waiting for the condition
Sep 17 16:43:06.862192 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:06.862085    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls podName:72346a52-84ef-4303-b475-17b6ea1f15d2 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:07.362063755 +0000 UTC m=+80.059565934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wv29x" (UID: "72346a52-84ef-4303-b475-17b6ea1f15d2") : failed to sync secret cache: timed out waiting for the condition
Sep 17 16:43:06.928971 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.928939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:43:06.936491 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:06.936429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkf5\" (UniqueName: \"kubernetes.io/projected/72346a52-84ef-4303-b475-17b6ea1f15d2-kube-api-access-qjkf5\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:07.041501 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:07.041318    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 16:43:07.373211 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:07.373135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:07.373364 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:07.373298    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:43:07.373431 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:07.373376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls podName:72346a52-84ef-4303-b475-17b6ea1f15d2 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:08.373354474 +0000 UTC m=+81.070856657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wv29x" (UID: "72346a52-84ef-4303-b475-17b6ea1f15d2") : secret "samples-operator-tls" not found
Sep 17 16:43:07.759328 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:07.759250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc" event={"ID":"5eef0184-be01-43ad-a56d-3f5819f04916","Type":"ContainerStarted","Data":"d096c19cbed57a1031938f5ab8acd68c96c4a85f71a21944f554e27dbf32d09e"}
Sep 17 16:43:07.775580 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:07.775525    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-lvfdc" podStartSLOduration=1.549806295 podStartE2EDuration="2.775506591s" totalImagesPullingTime="1.225700296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:07.774760802 +0000 UTC m=+80.472263005" watchObservedRunningTime="2026-09-17 16:43:07.775506591 +0000 UTC m=+80.473008798"
Sep 17 16:43:08.382219 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.382173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:08.382394 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:08.382301    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:43:08.382394 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:08.382385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls podName:72346a52-84ef-4303-b475-17b6ea1f15d2 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:10.382368423 +0000 UTC m=+83.079870603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wv29x" (UID: "72346a52-84ef-4303-b475-17b6ea1f15d2") : secret "samples-operator-tls" not found
Sep 17 16:43:08.763845 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.763795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" event={"ID":"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5","Type":"ContainerStarted","Data":"98e32c6a516d7bb1fb42e910175302f3f64dac7a599f799fdecbbfc947c9f3d2"}
Sep 17 16:43:08.764615 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.764171    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:08.765629 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.765521    2580 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-f9cwr container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.14:8443/readyz\": dial tcp 10.133.0.14:8443: connect: connection refused" start-of-body=
Sep 17 16:43:08.765629 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.765570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podUID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.14:8443/readyz\": dial tcp 10.133.0.14:8443: connect: connection refused"
Sep 17 16:43:08.766319 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.766274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4" event={"ID":"b50784ed-9d9d-440c-8075-b24d6b16a4fb","Type":"ContainerStarted","Data":"4a87a9560157b155b2ea2d0164d55778a95d54410ac95180061d66ef784a4d6b"}
Sep 17 16:43:08.791837 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.791788    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podStartSLOduration=1.426006319 podStartE2EDuration="3.791769826s" totalImagesPullingTime="2.365763507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:08.791635654 +0000 UTC m=+81.489137854" watchObservedRunningTime="2026-09-17 16:43:08.791769826 +0000 UTC m=+81.489272021"
Sep 17 16:43:08.810632 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:08.810593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4" podStartSLOduration=1.284365882 podStartE2EDuration="3.810581536s" totalImagesPullingTime="2.526215654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:08.809598898 +0000 UTC m=+81.507101095" watchObservedRunningTime="2026-09-17 16:43:08.810581536 +0000 UTC m=+81.508083734"
Sep 17 16:43:09.592706 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.592672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:09.592866 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.592725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:09.592866 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:09.592787    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:43:09.592866 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:09.592816    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:09.592866 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:09.592838    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:17.592824542 +0000 UTC m=+90.290326719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : configmap "kube-rbac-proxy" not found
Sep 17 16:43:09.593041 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:09.592878    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:17.592864407 +0000 UTC m=+90.290366583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:09.770616 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.770588    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/0.log"
Sep 17 16:43:09.771025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.770626    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5" containerID="98e32c6a516d7bb1fb42e910175302f3f64dac7a599f799fdecbbfc947c9f3d2" exitCode=255
Sep 17 16:43:09.771025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.770664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" event={"ID":"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5","Type":"ContainerDied","Data":"98e32c6a516d7bb1fb42e910175302f3f64dac7a599f799fdecbbfc947c9f3d2"}
Sep 17 16:43:09.771025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:09.771002    2580 scope.go:117] "RemoveContainer" containerID="98e32c6a516d7bb1fb42e910175302f3f64dac7a599f799fdecbbfc947c9f3d2"
Sep 17 16:43:10.401156 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.401114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:10.401336 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:10.401274    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:43:10.401381 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:10.401341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls podName:72346a52-84ef-4303-b475-17b6ea1f15d2 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:14.401324391 +0000 UTC m=+87.098826571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wv29x" (UID: "72346a52-84ef-4303-b475-17b6ea1f15d2") : secret "samples-operator-tls" not found
Sep 17 16:43:10.775710 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.775639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:43:10.776065 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.776035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/0.log"
Sep 17 16:43:10.776105 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.776063    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421" exitCode=255
Sep 17 16:43:10.776142 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.776133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" event={"ID":"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5","Type":"ContainerDied","Data":"73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"}
Sep 17 16:43:10.776186 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.776175    2580 scope.go:117] "RemoveContainer" containerID="98e32c6a516d7bb1fb42e910175302f3f64dac7a599f799fdecbbfc947c9f3d2"
Sep 17 16:43:10.776382 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:10.776363    2580 scope.go:117] "RemoveContainer" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"
Sep 17 16:43:10.776573 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:10.776550    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-f9cwr_openshift-console-operator(0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5)\"" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podUID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5"
Sep 17 16:43:11.345588 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.345554    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xglkx"]
Sep 17 16:43:11.349805 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.349788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.352384 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.352363    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 16:43:11.352490 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.352399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wnc85\""
Sep 17 16:43:11.353038 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.353025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 16:43:11.353331 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.353315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 16:43:11.353395 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.353350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 16:43:11.357168 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.357147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xglkx"]
Sep 17 16:43:11.407671 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.407650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-key\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.407769 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.407682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-cabundle\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.407769 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.407701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmgl\" (UniqueName: \"kubernetes.io/projected/209f975f-0d1b-4d49-844b-dfd42cf0b15b-kube-api-access-ncmgl\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.508193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.508175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-key\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.508273 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.508204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-cabundle\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.508273 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.508225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmgl\" (UniqueName: \"kubernetes.io/projected/209f975f-0d1b-4d49-844b-dfd42cf0b15b-kube-api-access-ncmgl\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.509004 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.508988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-cabundle\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.510714 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.510696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/209f975f-0d1b-4d49-844b-dfd42cf0b15b-signing-key\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.517295 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.517276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmgl\" (UniqueName: \"kubernetes.io/projected/209f975f-0d1b-4d49-844b-dfd42cf0b15b-kube-api-access-ncmgl\") pod \"service-ca-55f6fc7b86-xglkx\" (UID: \"209f975f-0d1b-4d49-844b-dfd42cf0b15b\") " pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.659092 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.659021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx"
Sep 17 16:43:11.780333 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.780306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:43:11.780651 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.780601    2580 scope.go:117] "RemoveContainer" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"
Sep 17 16:43:11.780757 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:11.780738    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-f9cwr_openshift-console-operator(0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5)\"" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podUID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5"
Sep 17 16:43:11.935644 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:11.935579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-xglkx"]
Sep 17 16:43:11.938358 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:11.938336    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209f975f_0d1b_4d49_844b_dfd42cf0b15b.slice/crio-352c9671563f7df5b30e0406d3e259f4a03e00375426e59fc1b9bdf68f43f987 WatchSource:0}: Error finding container 352c9671563f7df5b30e0406d3e259f4a03e00375426e59fc1b9bdf68f43f987: Status 404 returned error can't find the container with id 352c9671563f7df5b30e0406d3e259f4a03e00375426e59fc1b9bdf68f43f987
Sep 17 16:43:12.785808 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:12.785771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx" event={"ID":"209f975f-0d1b-4d49-844b-dfd42cf0b15b","Type":"ContainerStarted","Data":"352c9671563f7df5b30e0406d3e259f4a03e00375426e59fc1b9bdf68f43f987"}
Sep 17 16:43:14.431771 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:14.431726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:14.432182 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:14.431868    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 16:43:14.432182 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:14.431942    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls podName:72346a52-84ef-4303-b475-17b6ea1f15d2 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:22.431925419 +0000 UTC m=+95.129427613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wv29x" (UID: "72346a52-84ef-4303-b475-17b6ea1f15d2") : secret "samples-operator-tls" not found
Sep 17 16:43:14.793585 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:14.793554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx" event={"ID":"209f975f-0d1b-4d49-844b-dfd42cf0b15b","Type":"ContainerStarted","Data":"c35855f69afcaaafa259c5e3ae902dc67ae2a4d070bab6c9d53bfc4dee1477dd"}
Sep 17 16:43:14.814874 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:14.814821    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-xglkx" podStartSLOduration=1.831165838 podStartE2EDuration="3.814800866s" totalImagesPullingTime="1.983635028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:14.813170871 +0000 UTC m=+87.510673069" watchObservedRunningTime="2026-09-17 16:43:14.814800866 +0000 UTC m=+87.512303065"
Sep 17 16:43:16.038111 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:16.038077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:16.038575 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:16.038509    2580 scope.go:117] "RemoveContainer" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"
Sep 17 16:43:16.038720 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:16.038697    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-f9cwr_openshift-console-operator(0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5)\"" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podUID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5"
Sep 17 16:43:17.656455 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:17.656421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:17.656924 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:17.656548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:17.656924 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:17.656669    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:17.656924 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:17.656735    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls podName:99e8e5c3-cab1-4728-ab09-30f5987ca1bc nodeName:}" failed. No retries permitted until 2026-09-17 16:43:33.656717658 +0000 UTC m=+106.354219839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dmv8b" (UID: "99e8e5c3-cab1-4728-ab09-30f5987ca1bc") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:43:17.657037 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:17.656969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:18.764852 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:18.764817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:18.765277 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:18.765189    2580 scope.go:117] "RemoveContainer" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"
Sep 17 16:43:18.765352 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:43:18.765334    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-f9cwr_openshift-console-operator(0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5)\"" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" podUID="0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5"
Sep 17 16:43:22.499366 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:22.499327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:22.501868 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:22.501842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72346a52-84ef-4303-b475-17b6ea1f15d2-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wv29x\" (UID: \"72346a52-84ef-4303-b475-17b6ea1f15d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:22.513678 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:22.513654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"
Sep 17 16:43:22.788080 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:22.788057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x"]
Sep 17 16:43:23.823349 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:23.823279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x" event={"ID":"72346a52-84ef-4303-b475-17b6ea1f15d2","Type":"ContainerStarted","Data":"1626818f1ab4504cc1e7cc7b3d466440a9622a3037a590d4db5a621a20d32d4c"}
Sep 17 16:43:24.831920 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:24.831866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x" event={"ID":"72346a52-84ef-4303-b475-17b6ea1f15d2","Type":"ContainerStarted","Data":"df0aeb404e6b6299c2e2fe47504e85086ff0ac11e0c6f2ebe31ac352e44b6ce5"}
Sep 17 16:43:24.832278 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:24.831973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x" event={"ID":"72346a52-84ef-4303-b475-17b6ea1f15d2","Type":"ContainerStarted","Data":"d5ba4feeecca09b63f7064eaa3a4f7d9e18c822fdb480e08f2b1d9dd54b63007"}
Sep 17 16:43:24.858542 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:24.858496    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wv29x" podStartSLOduration=18.070494558 podStartE2EDuration="19.858480991s" totalImagesPullingTime="1.787986433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:24.857013266 +0000 UTC m=+97.554515470" watchObservedRunningTime="2026-09-17 16:43:24.858480991 +0000 UTC m=+97.555983191"
Sep 17 16:43:27.343256 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.343224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:43:27.343570 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.343284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:43:27.345622 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.345597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/095872e5-4dd2-4526-abec-91cea5f04f2d-metrics-tls\") pod \"dns-default-q49bh\" (UID: \"095872e5-4dd2-4526-abec-91cea5f04f2d\") " pod="openshift-dns/dns-default-q49bh"
Sep 17 16:43:27.345833 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.345814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"image-registry-69f6665948-pnfcj\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") " pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:43:27.368238 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.368189    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbjg6\""
Sep 17 16:43:27.372331 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.372317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:43:27.377764 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.377740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bdrbl\""
Sep 17 16:43:27.379852 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.379838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q49bh"
Sep 17 16:43:27.675563 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.675442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:43:27.698791 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.698768    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q49bh"]
Sep 17 16:43:27.701506 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:27.701472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095872e5_4dd2_4526_abec_91cea5f04f2d.slice/crio-9caf82b11f06d53074c74ee202477dbd015747c316a6844d280e0b4bbcbe58f7 WatchSource:0}: Error finding container 9caf82b11f06d53074c74ee202477dbd015747c316a6844d280e0b4bbcbe58f7: Status 404 returned error can't find the container with id 9caf82b11f06d53074c74ee202477dbd015747c316a6844d280e0b4bbcbe58f7
Sep 17 16:43:27.842391 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.842360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" event={"ID":"4cbb41c4-0eed-4850-99aa-f2ac252ad518","Type":"ContainerStarted","Data":"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"}
Sep 17 16:43:27.842492 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.842403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" event={"ID":"4cbb41c4-0eed-4850-99aa-f2ac252ad518","Type":"ContainerStarted","Data":"a25903179c2a99f07311d375b6cdea677e7edb024724274fb3bbc7fdd8488408"}
Sep 17 16:43:27.842492 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.842467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:43:27.843533 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.843513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49bh" event={"ID":"095872e5-4dd2-4526-abec-91cea5f04f2d","Type":"ContainerStarted","Data":"9caf82b11f06d53074c74ee202477dbd015747c316a6844d280e0b4bbcbe58f7"}
Sep 17 16:43:27.871477 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:27.871436    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" podStartSLOduration=97.871422307 podStartE2EDuration="1m37.871422307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:27.869881743 +0000 UTC m=+100.567383941" watchObservedRunningTime="2026-09-17 16:43:27.871422307 +0000 UTC m=+100.568924554"
Sep 17 16:43:29.852459 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:29.852432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49bh" event={"ID":"095872e5-4dd2-4526-abec-91cea5f04f2d","Type":"ContainerStarted","Data":"d31ec07ba46643baad4437407dc3ed6f929128217ddc5169bf22c7232b0d8d9b"}
Sep 17 16:43:30.358275 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.358246    2580 scope.go:117] "RemoveContainer" containerID="73bfbb0a0eaeaf1794c54dc321976dc2acfd9260d283f541bea677d0c143e421"
Sep 17 16:43:30.857350 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.857319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:43:30.857783 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.857412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr" event={"ID":"0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5","Type":"ContainerStarted","Data":"a2de199d6da2b779609413357edaef2dc44584f12a0affcc3c5209b577129f35"}
Sep 17 16:43:30.857843 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.857813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:30.859505 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.859483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q49bh" event={"ID":"095872e5-4dd2-4526-abec-91cea5f04f2d","Type":"ContainerStarted","Data":"e4e735404f1d5d7c80872aa18c87cda46489a47cbd66ed04f5010c9c9aa06cbf"}
Sep 17 16:43:30.859605 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.859595    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q49bh"
Sep 17 16:43:30.863164 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.863136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-f9cwr"
Sep 17 16:43:30.934229 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:30.934182    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q49bh" podStartSLOduration=66.03949051 podStartE2EDuration="1m7.93416773s" totalImagesPullingTime="1.89467722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:30.933844137 +0000 UTC m=+103.631346347" watchObservedRunningTime="2026-09-17 16:43:30.93416773 +0000 UTC m=+103.631669926"
Sep 17 16:43:31.735823 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:31.735795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jznpc"
Sep 17 16:43:33.696900 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:33.696861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:33.699257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:33.699238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/99e8e5c3-cab1-4728-ab09-30f5987ca1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dmv8b\" (UID: \"99e8e5c3-cab1-4728-ab09-30f5987ca1bc\") " pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:33.710366 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:33.710343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dmv8b"
Sep 17 16:43:34.016416 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:34.016335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dmv8b"]
Sep 17 16:43:34.021209 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:34.021184    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e8e5c3_cab1_4728_ab09_30f5987ca1bc.slice/crio-e02a80ae3a9ceecd89892c0337f5315368fa6b2014c1f5cc8781b0ce7f4015e9 WatchSource:0}: Error finding container e02a80ae3a9ceecd89892c0337f5315368fa6b2014c1f5cc8781b0ce7f4015e9: Status 404 returned error can't find the container with id e02a80ae3a9ceecd89892c0337f5315368fa6b2014c1f5cc8781b0ce7f4015e9
Sep 17 16:43:34.873598 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:34.873572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmv8b" event={"ID":"99e8e5c3-cab1-4728-ab09-30f5987ca1bc","Type":"ContainerStarted","Data":"ee94869ab0add8d2c5227704ad9c2fff4dc8929b76cf2f13c8e28db21b0df809"}
Sep 17 16:43:34.873927 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:34.873644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmv8b" event={"ID":"99e8e5c3-cab1-4728-ab09-30f5987ca1bc","Type":"ContainerStarted","Data":"e02a80ae3a9ceecd89892c0337f5315368fa6b2014c1f5cc8781b0ce7f4015e9"}
Sep 17 16:43:35.281482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.281394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"]
Sep 17 16:43:35.284023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.284004    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"]
Sep 17 16:43:35.284160 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.284144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.285884 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.285862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.288177 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.288158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 16:43:35.288429 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.288415    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 16:43:35.288513 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.288418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 16:43:35.288994 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.288969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 16:43:35.288994 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.288990    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 16:43:35.290259 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.290240    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 16:43:35.290388 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.290289    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 16:43:35.290388 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.290294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 16:43:35.306114 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.306092    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"]
Sep 17 16:43:35.309198 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.309176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"]
Sep 17 16:43:35.311796 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.311776    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:43:35.405315 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.405286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cb59b98fb-4jxhj"]
Sep 17 16:43:35.407418 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.407639 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t89s\" (UniqueName: \"kubernetes.io/projected/e1aea05f-2e14-4b10-9614-788c91903cdd-kube-api-access-4t89s\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.407689 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9kd\" (UniqueName: \"kubernetes.io/projected/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-kube-api-access-mp9kd\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.407689 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.407769 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e1aea05f-2e14-4b10-9614-788c91903cdd-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.407769 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.407840 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.407840 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1aea05f-2e14-4b10-9614-788c91903cdd-tmp\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.407840 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.407967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.407870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.465577 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.465555    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb59b98fb-4jxhj"]
Sep 17 16:43:35.508534 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9kd\" (UniqueName: \"kubernetes.io/projected/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-kube-api-access-mp9kd\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.508669 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-certificates\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.508669 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.508669 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e1aea05f-2e14-4b10-9614-788c91903cdd-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.508669 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.508669 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1aea05f-2e14-4b10-9614-788c91903cdd-tmp\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/549fa0e2-b96d-4c2d-bb79-713e482c6c06-ca-trust-extracted\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-tls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.508845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-image-registry-private-configuration\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-installation-pull-secrets\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-bound-sa-token\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-trusted-ca\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1aea05f-2e14-4b10-9614-788c91903cdd-tmp\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.509154 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-kube-api-access-gcgls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.509942 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t89s\" (UniqueName: \"kubernetes.io/projected/e1aea05f-2e14-4b10-9614-788c91903cdd-kube-api-access-4t89s\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.510011 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.509991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.511929 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.511874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e1aea05f-2e14-4b10-9614-788c91903cdd-klusterlet-config\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.512023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.511984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-ca\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.512185 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.512164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.512310 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.512278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-hub\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.512572 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.512552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.539252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.539216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t89s\" (UniqueName: \"kubernetes.io/projected/e1aea05f-2e14-4b10-9614-788c91903cdd-kube-api-access-4t89s\") pod \"klusterlet-addon-workmgr-85f689997d-z98rg\" (UID: \"e1aea05f-2e14-4b10-9614-788c91903cdd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.540797 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.540775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9kd\" (UniqueName: \"kubernetes.io/projected/b76f11e6-2ac6-49b8-ad4f-1f81956a2eba-kube-api-access-mp9kd\") pod \"cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2\" (UID: \"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.608798 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.608764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"
Sep 17 16:43:35.610048 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-bound-sa-token\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610151 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-trusted-ca\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610151 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-kube-api-access-gcgls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610151 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-certificates\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610302 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/549fa0e2-b96d-4c2d-bb79-713e482c6c06-ca-trust-extracted\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610302 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-tls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610302 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-image-registry-private-configuration\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610442 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-installation-pull-secrets\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610687 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/549fa0e2-b96d-4c2d-bb79-713e482c6c06-ca-trust-extracted\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610986 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-certificates\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.610986 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.610977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/549fa0e2-b96d-4c2d-bb79-713e482c6c06-trusted-ca\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.613021 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.612999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-image-registry-private-configuration\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.613209 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.613191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/549fa0e2-b96d-4c2d-bb79-713e482c6c06-installation-pull-secrets\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.613284 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.613259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-registry-tls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.615057 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.615039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:35.620300 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.620279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-bound-sa-token\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.623063 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.623044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgls\" (UniqueName: \"kubernetes.io/projected/549fa0e2-b96d-4c2d-bb79-713e482c6c06-kube-api-access-gcgls\") pod \"image-registry-6cb59b98fb-4jxhj\" (UID: \"549fa0e2-b96d-4c2d-bb79-713e482c6c06\") " pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.715708 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.715668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:35.887291 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.887219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmv8b" event={"ID":"99e8e5c3-cab1-4728-ab09-30f5987ca1bc","Type":"ContainerStarted","Data":"e73aa3b45cba154c85ea0017c9b728758807f01a6c8834353ca589089f0e26c7"}
Sep 17 16:43:35.903358 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.903301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2"]
Sep 17 16:43:35.907524 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:35.907498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb76f11e6_2ac6_49b8_ad4f_1f81956a2eba.slice/crio-b37dc118438ca51193f417f6e3c4039ba90fc7cb2833028d3f26dc13b8b0fc3f WatchSource:0}: Error finding container b37dc118438ca51193f417f6e3c4039ba90fc7cb2833028d3f26dc13b8b0fc3f: Status 404 returned error can't find the container with id b37dc118438ca51193f417f6e3c4039ba90fc7cb2833028d3f26dc13b8b0fc3f
Sep 17 16:43:35.909286 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:35.909262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"]
Sep 17 16:43:35.912791 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:35.912761    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1aea05f_2e14_4b10_9614_788c91903cdd.slice/crio-35ff76dece6c712ef742598b76fff789d7336a5d7270a48d0dd388c6d4e06455 WatchSource:0}: Error finding container 35ff76dece6c712ef742598b76fff789d7336a5d7270a48d0dd388c6d4e06455: Status 404 returned error can't find the container with id 35ff76dece6c712ef742598b76fff789d7336a5d7270a48d0dd388c6d4e06455
Sep 17 16:43:36.001341 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.001309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb59b98fb-4jxhj"]
Sep 17 16:43:36.358554 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:36.358509    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549fa0e2_b96d_4c2d_bb79_713e482c6c06.slice/crio-da7370c1f062417274b3381df8098e1975d10a21f7107e56c631180211880236 WatchSource:0}: Error finding container da7370c1f062417274b3381df8098e1975d10a21f7107e56c631180211880236: Status 404 returned error can't find the container with id da7370c1f062417274b3381df8098e1975d10a21f7107e56c631180211880236
Sep 17 16:43:36.898453 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.897181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dmv8b" event={"ID":"99e8e5c3-cab1-4728-ab09-30f5987ca1bc","Type":"ContainerStarted","Data":"c523207e8882c4d74e565f18dabf0a6dec9b8804fd45fef1655d3e75f3f13064"}
Sep 17 16:43:36.902884 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.902808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj" event={"ID":"549fa0e2-b96d-4c2d-bb79-713e482c6c06","Type":"ContainerStarted","Data":"08346bc59612d9cbc7353c4c9941fbfa8d8a4e91678ed31b3692faa428c98707"}
Sep 17 16:43:36.902884 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.902860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj" event={"ID":"549fa0e2-b96d-4c2d-bb79-713e482c6c06","Type":"ContainerStarted","Data":"da7370c1f062417274b3381df8098e1975d10a21f7107e56c631180211880236"}
Sep 17 16:43:36.903611 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.903397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:36.906077 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.906043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2" event={"ID":"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba","Type":"ContainerStarted","Data":"b37dc118438ca51193f417f6e3c4039ba90fc7cb2833028d3f26dc13b8b0fc3f"}
Sep 17 16:43:36.908007 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.907949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg" event={"ID":"e1aea05f-2e14-4b10-9614-788c91903cdd","Type":"ContainerStarted","Data":"35ff76dece6c712ef742598b76fff789d7336a5d7270a48d0dd388c6d4e06455"}
Sep 17 16:43:36.956271 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.956211    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dmv8b" podStartSLOduration=33.751043456 podStartE2EDuration="35.956192182s" totalImagesPullingTime="2.205148726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:36.929323039 +0000 UTC m=+109.626825239" watchObservedRunningTime="2026-09-17 16:43:36.956192182 +0000 UTC m=+109.653694380"
Sep 17 16:43:36.956927 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:36.956865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj" podStartSLOduration=1.956852265 podStartE2EDuration="1.956852265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:36.955743631 +0000 UTC m=+109.653245830" watchObservedRunningTime="2026-09-17 16:43:36.956852265 +0000 UTC m=+109.654354462"
Sep 17 16:43:40.865234 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:40.865200    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q49bh"
Sep 17 16:43:41.928936 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.928824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg" event={"ID":"e1aea05f-2e14-4b10-9614-788c91903cdd","Type":"ContainerStarted","Data":"7858ce7195b5a2401ad3719407cc97c0029ebab72785ee16c0e73a7fd001e5ef"}
Sep 17 16:43:41.928936 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.928889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:41.930590 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.930568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg"
Sep 17 16:43:41.931250 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.931224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2" event={"ID":"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba","Type":"ContainerStarted","Data":"942c093ce27b83b996c711560e6cd326e0e682095b445519f3dfcd7d1a218206"}
Sep 17 16:43:41.931341 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.931268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2" event={"ID":"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba","Type":"ContainerStarted","Data":"5b16c0009e408ff7d8d7a29c0314170161bd2475ab47fcfcdc2e445e70f6ab5a"}
Sep 17 16:43:41.931341 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.931285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2" event={"ID":"b76f11e6-2ac6-49b8-ad4f-1f81956a2eba","Type":"ContainerStarted","Data":"5f31d0d2d0e440be7da12ddd8f4693598a87eddc206c59c0c7e2059fb613c055"}
Sep 17 16:43:41.947311 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.947268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85f689997d-z98rg" podStartSLOduration=1.306864219 podStartE2EDuration="6.947253975s" totalImagesPullingTime="5.640389756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:41.946968251 +0000 UTC m=+114.644470449" watchObservedRunningTime="2026-09-17 16:43:41.947253975 +0000 UTC m=+114.644756166"
Sep 17 16:43:41.968700 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:41.968656    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75dd6bd9c9-7r7p2" podStartSLOduration=1.328110114 podStartE2EDuration="6.968644877s" totalImagesPullingTime="5.640534763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:41.968069574 +0000 UTC m=+114.665571772" watchObservedRunningTime="2026-09-17 16:43:41.968644877 +0000 UTC m=+114.666147075"
Sep 17 16:43:45.319024 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:45.318994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:43:48.931477 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.931440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fpnz9"]
Sep 17 16:43:48.936064 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.936047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:48.938377 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.938355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:48.938501 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.938400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:43:48.938501 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.938399    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:48.938991 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.938969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:48.939100 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.938973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:43:48.939290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.939269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:43:48.939290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:48.939272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-48wqc\""
Sep 17 16:43:49.027247 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-tls\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027412 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027412 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-root\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027412 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-metrics-client-ca\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027412 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-sys\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027412 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4tq\" (UniqueName: \"kubernetes.io/projected/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-kube-api-access-jl4tq\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027575 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-wtmp\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027575 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-textfile\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.027575 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.027531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.128744 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.128744 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-root\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-metrics-client-ca\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-sys\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4tq\" (UniqueName: \"kubernetes.io/projected/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-kube-api-access-jl4tq\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-wtmp\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-root\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-textfile\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-tls\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.128968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-sys\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129010 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.129011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-wtmp\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129503 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.129292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-textfile\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129503 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.129370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-accelerators-collector-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.129503 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.129497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-metrics-client-ca\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.131468 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.131449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-tls\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.131571 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.131555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.139631 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.139597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4tq\" (UniqueName: \"kubernetes.io/projected/6f103dfe-af4d-4c2c-8d68-64f60caa6ab7-kube-api-access-jl4tq\") pod \"node-exporter-fpnz9\" (UID: \"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7\") " pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.246563 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.246492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fpnz9"
Sep 17 16:43:49.259360 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:49.259337    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f103dfe_af4d_4c2c_8d68_64f60caa6ab7.slice/crio-06c36b41a7fc02425f4645e7f8a18fe8bfae8ff5af5c97bd9ac512a4f0e101ee WatchSource:0}: Error finding container 06c36b41a7fc02425f4645e7f8a18fe8bfae8ff5af5c97bd9ac512a4f0e101ee: Status 404 returned error can't find the container with id 06c36b41a7fc02425f4645e7f8a18fe8bfae8ff5af5c97bd9ac512a4f0e101ee
Sep 17 16:43:49.964268 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:49.964232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpnz9" event={"ID":"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7","Type":"ContainerStarted","Data":"06c36b41a7fc02425f4645e7f8a18fe8bfae8ff5af5c97bd9ac512a4f0e101ee"}
Sep 17 16:43:50.969266 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:50.969226    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f103dfe-af4d-4c2c-8d68-64f60caa6ab7" containerID="e1c9f3bf4e8547b1ba3156fdc2ab95531c17c30df1a2de6918d7b9dbac2a45bd" exitCode=0
Sep 17 16:43:50.969635 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:50.969285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpnz9" event={"ID":"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7","Type":"ContainerDied","Data":"e1c9f3bf4e8547b1ba3156fdc2ab95531c17c30df1a2de6918d7b9dbac2a45bd"}
Sep 17 16:43:51.975767 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:51.975729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpnz9" event={"ID":"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7","Type":"ContainerStarted","Data":"358ad456a702b12586782ae49860c04d0c4945a3a46a383e412813fd600e4eab"}
Sep 17 16:43:51.976167 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:51.975797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fpnz9" event={"ID":"6f103dfe-af4d-4c2c-8d68-64f60caa6ab7","Type":"ContainerStarted","Data":"8ffc648eb6b042ecce3097373c6df97cb31c449de7ec5b0e34086bba9ce75b48"}
Sep 17 16:43:52.001376 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:52.001331    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fpnz9" podStartSLOduration=2.9224346519999997 podStartE2EDuration="4.001317415s" totalImagesPullingTime="1.012492009s" totalInitContainerRuntime="66.390754ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:52.000002243 +0000 UTC m=+124.697504468" watchObservedRunningTime="2026-09-17 16:43:52.001317415 +0000 UTC m=+124.698819657"
Sep 17 16:43:53.623346 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.623317    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"]
Sep 17 16:43:53.626428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.626408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.628889 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.628869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 16:43:53.628889 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.628881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6tqxd\""
Sep 17 16:43:53.638107 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.638083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"]
Sep 17 16:43:53.767176 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.767148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.767311 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.767208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56fb\" (UniqueName: \"kubernetes.io/projected/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-kube-api-access-l56fb\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.868469 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.868439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.868632 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.868497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56fb\" (UniqueName: \"kubernetes.io/projected/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-kube-api-access-l56fb\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.871088 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.871062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.886480 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.886419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56fb\" (UniqueName: \"kubernetes.io/projected/c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec-kube-api-access-l56fb\") pod \"monitoring-plugin-5984459ff8-lktl5\" (UID: \"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:53.935370 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:53.935343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:54.204580 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.204511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"]
Sep 17 16:43:54.207499 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:54.207470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53fff2f_d07b_4ecf_a1b8_3d55e635f7ec.slice/crio-8a6e1e0f665cd3081b2fa453e6c0ad7593b13aafbe643e94b4a1f1578f8a43a8 WatchSource:0}: Error finding container 8a6e1e0f665cd3081b2fa453e6c0ad7593b13aafbe643e94b4a1f1578f8a43a8: Status 404 returned error can't find the container with id 8a6e1e0f665cd3081b2fa453e6c0ad7593b13aafbe643e94b4a1f1578f8a43a8
Sep 17 16:43:54.507672 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.507595    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:43:54.511917 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.511894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.514280 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.514249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:43:54.515049 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:43:54.515166 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:43:54.515166 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:43:54.515338 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515322    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:43:54.515418 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5b6h2\""
Sep 17 16:43:54.515465 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:43:54.515499 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.515452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:43:54.520277 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.520251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 16:43:54.521831 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.521808    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:43:54.575402 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575526 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575526 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6czt\" (UniqueName: \"kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575592 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575626 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575626 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.575690 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.575638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.676664 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6czt\" (UniqueName: \"kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677106 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.676813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677605 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.677580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677605 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.677590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677732 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.677690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.677849 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.677828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.679227 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.679207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.679343 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.679327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.684323 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.684304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6czt\" (UniqueName: \"kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt\") pod \"console-599c77ccb8-cm48r\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") " pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.823613 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.823572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:43:54.990042 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:54.990000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5" event={"ID":"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec","Type":"ContainerStarted","Data":"8a6e1e0f665cd3081b2fa453e6c0ad7593b13aafbe643e94b4a1f1578f8a43a8"}
Sep 17 16:43:55.203145 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:55.203116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:43:55.205330 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:55.205303    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00e52d1_6434_4558_b5f9_1c3fce36455e.slice/crio-a69792264964ad804805e7566925a45031bd532639bd95d0aea1f87db9e3b469 WatchSource:0}: Error finding container a69792264964ad804805e7566925a45031bd532639bd95d0aea1f87db9e3b469: Status 404 returned error can't find the container with id a69792264964ad804805e7566925a45031bd532639bd95d0aea1f87db9e3b469
Sep 17 16:43:55.996532 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:55.996501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599c77ccb8-cm48r" event={"ID":"f00e52d1-6434-4558-b5f9-1c3fce36455e","Type":"ContainerStarted","Data":"a69792264964ad804805e7566925a45031bd532639bd95d0aea1f87db9e3b469"}
Sep 17 16:43:57.003271 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:57.003235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5" event={"ID":"c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec","Type":"ContainerStarted","Data":"afde8e5d8368f789519859ce62aacf2ecdb6a30cbebf418387d393e8ceaabf88"}
Sep 17 16:43:57.003650 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:57.003637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:57.009509 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:57.009484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5"
Sep 17 16:43:57.038039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:57.037983    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-lktl5" podStartSLOduration=2.316966858 podStartE2EDuration="4.037961829s" totalImagesPullingTime="1.720994971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:57.020748939 +0000 UTC m=+129.718251139" watchObservedRunningTime="2026-09-17 16:43:57.037961829 +0000 UTC m=+129.735464027"
Sep 17 16:43:58.923034 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:58.923002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cb59b98fb-4jxhj"
Sep 17 16:43:59.223967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:59.223869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:43:59.226499 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:59.226471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336b6a4-be76-4751-92fa-9372275f0ddd-metrics-certs\") pod \"network-metrics-daemon-k8mwq\" (UID: \"9336b6a4-be76-4751-92fa-9372275f0ddd\") " pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:43:59.275615 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:59.275590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n5btm\""
Sep 17 16:43:59.283452 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:59.283431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k8mwq"
Sep 17 16:43:59.711311 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:43:59.711279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k8mwq"]
Sep 17 16:43:59.715557 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:43:59.715525    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9336b6a4_be76_4751_92fa_9372275f0ddd.slice/crio-c1960ec87fb71de47899188509d87637f231955956cf4c0c0d4b23bff0c72f92 WatchSource:0}: Error finding container c1960ec87fb71de47899188509d87637f231955956cf4c0c0d4b23bff0c72f92: Status 404 returned error can't find the container with id c1960ec87fb71de47899188509d87637f231955956cf4c0c0d4b23bff0c72f92
Sep 17 16:44:00.018604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.018558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599c77ccb8-cm48r" event={"ID":"f00e52d1-6434-4558-b5f9-1c3fce36455e","Type":"ContainerStarted","Data":"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"}
Sep 17 16:44:00.019756 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.019728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k8mwq" event={"ID":"9336b6a4-be76-4751-92fa-9372275f0ddd","Type":"ContainerStarted","Data":"c1960ec87fb71de47899188509d87637f231955956cf4c0c0d4b23bff0c72f92"}
Sep 17 16:44:00.044854 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.044807    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599c77ccb8-cm48r" podStartSLOduration=1.751675806 podStartE2EDuration="6.044792283s" totalImagesPullingTime="4.293116477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:00.044191176 +0000 UTC m=+132.741693376" watchObservedRunningTime="2026-09-17 16:44:00.044792283 +0000 UTC m=+132.742294482"
Sep 17 16:44:00.332283 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.332210    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" podUID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" containerName="registry" containerID="cri-o://93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d" gracePeriod=30
Sep 17 16:44:00.743638 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.743613    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:44:00.837670 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.837670 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.837843 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.837843 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.837843 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.837843 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.838142 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r77c\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.838142 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.837885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets\") pod \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\" (UID: \"4cbb41c4-0eed-4850-99aa-f2ac252ad518\") "
Sep 17 16:44:00.838249 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.838175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:00.838249 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.838197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:00.840778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.840643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:00.840934 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.840863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:00.841008 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.840924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:00.841548 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.841505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c" (OuterVolumeSpecName: "kube-api-access-8r77c") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "kube-api-access-8r77c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:00.841548 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.841513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:00.847276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.847249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4cbb41c4-0eed-4850-99aa-f2ac252ad518" (UID: "4cbb41c4-0eed-4850-99aa-f2ac252ad518"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:00.938719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938685    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938713    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-image-registry-private-configuration\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938719 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938723    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r77c\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-kube-api-access-8r77c\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938951 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938732    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbb41c4-0eed-4850-99aa-f2ac252ad518-installation-pull-secrets\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938951 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938741    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbb41c4-0eed-4850-99aa-f2ac252ad518-bound-sa-token\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938951 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938750    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbb41c4-0eed-4850-99aa-f2ac252ad518-ca-trust-extracted\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938951 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938758    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-trusted-ca\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:00.938951 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:00.938766    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbb41c4-0eed-4850-99aa-f2ac252ad518-registry-certificates\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:01.025394 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.025362    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" containerID="93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d" exitCode=0
Sep 17 16:44:01.025772 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.025408    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f6665948-pnfcj"
Sep 17 16:44:01.025772 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.025454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" event={"ID":"4cbb41c4-0eed-4850-99aa-f2ac252ad518","Type":"ContainerDied","Data":"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"}
Sep 17 16:44:01.025772 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.025509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f6665948-pnfcj" event={"ID":"4cbb41c4-0eed-4850-99aa-f2ac252ad518","Type":"ContainerDied","Data":"a25903179c2a99f07311d375b6cdea677e7edb024724274fb3bbc7fdd8488408"}
Sep 17 16:44:01.025772 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.025535    2580 scope.go:117] "RemoveContainer" containerID="93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"
Sep 17 16:44:01.027680 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.027646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k8mwq" event={"ID":"9336b6a4-be76-4751-92fa-9372275f0ddd","Type":"ContainerStarted","Data":"d9376d8364d2611105bbc1b9b0fd1031c1845802526639ce03b435a9cfca5020"}
Sep 17 16:44:01.027764 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.027699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k8mwq" event={"ID":"9336b6a4-be76-4751-92fa-9372275f0ddd","Type":"ContainerStarted","Data":"d5eb4bb4fe8fd7c0c0ad2e48cd483d6b89fd83d1ca625067cbcb635eb7241a94"}
Sep 17 16:44:01.034461 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.034446    2580 scope.go:117] "RemoveContainer" containerID="93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"
Sep 17 16:44:01.034753 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:44:01.034735    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d\": container with ID starting with 93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d not found: ID does not exist" containerID="93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"
Sep 17 16:44:01.034795 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.034761    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d"} err="failed to get container status \"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d\": rpc error: code = NotFound desc = could not find container \"93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d\": container with ID starting with 93273d1daadba01448eff0dc7b086420a7b72bc366981fb5c4894d32af9e6e8d not found: ID does not exist"
Sep 17 16:44:01.087857 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.087814    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k8mwq" podStartSLOduration=131.171858406 podStartE2EDuration="2m12.087801165s" totalImagesPullingTime="915.942759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:01.085872213 +0000 UTC m=+133.783374411" watchObservedRunningTime="2026-09-17 16:44:01.087801165 +0000 UTC m=+133.785303363"
Sep 17 16:44:01.110926 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.110855    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:44:01.115305 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.115283    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69f6665948-pnfcj"]
Sep 17 16:44:01.363169 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:01.363090    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" path="/var/lib/kubelet/pods/4cbb41c4-0eed-4850-99aa-f2ac252ad518/volumes"
Sep 17 16:44:03.309382 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.309351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:44:03.357690 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.357660    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:44:03.357967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.357954    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" containerName="registry"
Sep 17 16:44:03.358020 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.357969    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" containerName="registry"
Sep 17 16:44:03.358054 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.358031    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbb41c4-0eed-4850-99aa-f2ac252ad518" containerName="registry"
Sep 17 16:44:03.362097 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.362070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.379952 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.379926    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:44:03.458724 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458838 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458838 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lmm\" (UniqueName: \"kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458838 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.458967 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.458879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.559766 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.559766 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lmm\" (UniqueName: \"kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.559766 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560023 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.559885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560508 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.560486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560734 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.560605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560785 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.560668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.560785 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.560726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.562309 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.562285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.562500 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.562480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.567866 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.567839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lmm\" (UniqueName: \"kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm\") pod \"console-764dc994d7-bcn42\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") " pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.671549 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.671521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:03.944778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:03.944705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:44:03.947405 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:44:03.947377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a64c0c_4fee_4995_b323_2532786c4044.slice/crio-8b47c46021593b2482bb11eeff66c3176472c4dbf92c01e145dbefcb1f1eebd2 WatchSource:0}: Error finding container 8b47c46021593b2482bb11eeff66c3176472c4dbf92c01e145dbefcb1f1eebd2: Status 404 returned error can't find the container with id 8b47c46021593b2482bb11eeff66c3176472c4dbf92c01e145dbefcb1f1eebd2
Sep 17 16:44:04.043769 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:04.043741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764dc994d7-bcn42" event={"ID":"35a64c0c-4fee-4995-b323-2532786c4044","Type":"ContainerStarted","Data":"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"}
Sep 17 16:44:04.043895 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:04.043788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764dc994d7-bcn42" event={"ID":"35a64c0c-4fee-4995-b323-2532786c4044","Type":"ContainerStarted","Data":"8b47c46021593b2482bb11eeff66c3176472c4dbf92c01e145dbefcb1f1eebd2"}
Sep 17 16:44:04.063881 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:04.063823    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-764dc994d7-bcn42" podStartSLOduration=1.063802012 podStartE2EDuration="1.063802012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:04.062969797 +0000 UTC m=+136.760471996" watchObservedRunningTime="2026-09-17 16:44:04.063802012 +0000 UTC m=+136.761304212"
Sep 17 16:44:04.824290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:04.824260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:44:07.056002 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:07.055965    2580 generic.go:356] "Generic (PLEG): container finished" podID="6073644f-97cc-4086-a8da-3b606a68d09b" containerID="4e4feb649dc5486ee59530f814aae69768c483fdb3a8589ecb1edff06e33ce6c" exitCode=0
Sep 17 16:44:07.056470 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:07.056045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8" event={"ID":"6073644f-97cc-4086-a8da-3b606a68d09b","Type":"ContainerDied","Data":"4e4feb649dc5486ee59530f814aae69768c483fdb3a8589ecb1edff06e33ce6c"}
Sep 17 16:44:07.056542 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:07.056479    2580 scope.go:117] "RemoveContainer" containerID="4e4feb649dc5486ee59530f814aae69768c483fdb3a8589ecb1edff06e33ce6c"
Sep 17 16:44:07.818731 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:07.818699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5hds_32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7/serve-healthcheck-canary/0.log"
Sep 17 16:44:08.062379 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:08.062347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sd2q8" event={"ID":"6073644f-97cc-4086-a8da-3b606a68d09b","Type":"ContainerStarted","Data":"f3abe7a6328f03789436c7ebf073def01870eb7f753ef47190ac7ff1e321f3cb"}
Sep 17 16:44:13.672607 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:13.672572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:13.673070 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:13.672637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:13.677435 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:13.677415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:14.089182 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:14.089151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:44:25.129404 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:25.129370    2580 generic.go:356] "Generic (PLEG): container finished" podID="b50784ed-9d9d-440c-8075-b24d6b16a4fb" containerID="4a87a9560157b155b2ea2d0164d55778a95d54410ac95180061d66ef784a4d6b" exitCode=0
Sep 17 16:44:25.129404 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:25.129409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4" event={"ID":"b50784ed-9d9d-440c-8075-b24d6b16a4fb","Type":"ContainerDied","Data":"4a87a9560157b155b2ea2d0164d55778a95d54410ac95180061d66ef784a4d6b"}
Sep 17 16:44:25.129835 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:25.129692    2580 scope.go:117] "RemoveContainer" containerID="4a87a9560157b155b2ea2d0164d55778a95d54410ac95180061d66ef784a4d6b"
Sep 17 16:44:26.135551 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:26.135520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mskw4" event={"ID":"b50784ed-9d9d-440c-8075-b24d6b16a4fb","Type":"ContainerStarted","Data":"ebbcc4f96ed6dccd63f522d1d213669095861be23bb88bc18b53efc1588fb912"}
Sep 17 16:44:28.329450 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.329389    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599c77ccb8-cm48r" podUID="f00e52d1-6434-4558-b5f9-1c3fce36455e" containerName="console" containerID="cri-o://44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9" gracePeriod=15
Sep 17 16:44:28.633890 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.633864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:44:28.654128 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654100    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654257 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6czt\" (UniqueName: \"kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654514 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle\") pod \"f00e52d1-6434-4558-b5f9-1c3fce36455e\" (UID: \"f00e52d1-6434-4558-b5f9-1c3fce36455e\") "
Sep 17 16:44:28.654604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:28.654703 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:28.654703 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config" (OuterVolumeSpecName: "console-config") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:28.654808 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.654735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:28.656618 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.656591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:28.656706 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.656628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:28.656706 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.656697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt" (OuterVolumeSpecName: "kube-api-access-s6czt") pod "f00e52d1-6434-4558-b5f9-1c3fce36455e" (UID: "f00e52d1-6434-4558-b5f9-1c3fce36455e"). InnerVolumeSpecName "kube-api-access-s6czt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:28.754866 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754841    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.754866 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754863    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.755030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754873    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-service-ca\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.755030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754881    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f00e52d1-6434-4558-b5f9-1c3fce36455e-console-oauth-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.755030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754890    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6czt\" (UniqueName: \"kubernetes.io/projected/f00e52d1-6434-4558-b5f9-1c3fce36455e-kube-api-access-s6czt\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.755030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754898    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-trusted-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:28.755030 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:28.754925    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f00e52d1-6434-4558-b5f9-1c3fce36455e-oauth-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:44:29.149430 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.149401    2580 generic.go:356] "Generic (PLEG): container finished" podID="f00e52d1-6434-4558-b5f9-1c3fce36455e" containerID="44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9" exitCode=0
Sep 17 16:44:29.149604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.149442    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599c77ccb8-cm48r"
Sep 17 16:44:29.149604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.149441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599c77ccb8-cm48r" event={"ID":"f00e52d1-6434-4558-b5f9-1c3fce36455e","Type":"ContainerDied","Data":"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"}
Sep 17 16:44:29.149604 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.149582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599c77ccb8-cm48r" event={"ID":"f00e52d1-6434-4558-b5f9-1c3fce36455e","Type":"ContainerDied","Data":"a69792264964ad804805e7566925a45031bd532639bd95d0aea1f87db9e3b469"}
Sep 17 16:44:29.149774 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.149616    2580 scope.go:117] "RemoveContainer" containerID="44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"
Sep 17 16:44:29.158375 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.158360    2580 scope.go:117] "RemoveContainer" containerID="44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"
Sep 17 16:44:29.158651 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:44:29.158632    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9\": container with ID starting with 44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9 not found: ID does not exist" containerID="44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"
Sep 17 16:44:29.158696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.158659    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9"} err="failed to get container status \"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9\": rpc error: code = NotFound desc = could not find container \"44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9\": container with ID starting with 44290ff5c8eaa7f63a04f8533ec4a90df882651241157947643e5331176aece9 not found: ID does not exist"
Sep 17 16:44:29.175281 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.175237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:44:29.177953 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.177934    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599c77ccb8-cm48r"]
Sep 17 16:44:29.364083 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:44:29.364050    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00e52d1-6434-4558-b5f9-1c3fce36455e" path="/var/lib/kubelet/pods/f00e52d1-6434-4558-b5f9-1c3fce36455e/volumes"
Sep 17 16:45:06.414859 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.414819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:45:06.415349 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.415100    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00e52d1-6434-4558-b5f9-1c3fce36455e" containerName="console"
Sep 17 16:45:06.415349 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.415112    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00e52d1-6434-4558-b5f9-1c3fce36455e" containerName="console"
Sep 17 16:45:06.415349 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.415159    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00e52d1-6434-4558-b5f9-1c3fce36455e" containerName="console"
Sep 17 16:45:06.418173 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.418148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.439895 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.439874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:45:06.522657 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522657 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw96\" (UniqueName: \"kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522812 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522812 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522812 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522929 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.522929 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.522857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623276 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw96\" (UniqueName: \"kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.623444 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.623438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.624230 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.624144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.624230 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.624167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.624230 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.624191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.624704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.624680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.625889 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.625869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.626176 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.626158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.632749 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.632729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw96\" (UniqueName: \"kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96\") pod \"console-7568d66fcb-64vb9\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") " pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:06.727582 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:06.727505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:07.007949 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:07.007721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:45:07.010574 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:45:07.010545    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff95c8fb_e35d_4f49_a8f1_61814b94cd68.slice/crio-24e047e5afc69c07dcfe180ce1dbcb404d2501e71039a795d2a0080d00d91858 WatchSource:0}: Error finding container 24e047e5afc69c07dcfe180ce1dbcb404d2501e71039a795d2a0080d00d91858: Status 404 returned error can't find the container with id 24e047e5afc69c07dcfe180ce1dbcb404d2501e71039a795d2a0080d00d91858
Sep 17 16:45:07.289157 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:07.289121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7568d66fcb-64vb9" event={"ID":"ff95c8fb-e35d-4f49-a8f1-61814b94cd68","Type":"ContainerStarted","Data":"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"}
Sep 17 16:45:07.289315 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:07.289171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7568d66fcb-64vb9" event={"ID":"ff95c8fb-e35d-4f49-a8f1-61814b94cd68","Type":"ContainerStarted","Data":"24e047e5afc69c07dcfe180ce1dbcb404d2501e71039a795d2a0080d00d91858"}
Sep 17 16:45:07.314411 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:07.314369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7568d66fcb-64vb9" podStartSLOduration=1.314354933 podStartE2EDuration="1.314354933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:07.312355324 +0000 UTC m=+200.009857522" watchObservedRunningTime="2026-09-17 16:45:07.314354933 +0000 UTC m=+200.011857130"
Sep 17 16:45:16.728261 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:16.728222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:16.728650 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:16.728292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:16.734004 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:16.733982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:17.328305 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:17.328280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:45:17.386161 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:17.386132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:45:42.412662 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.412608    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-764dc994d7-bcn42" podUID="35a64c0c-4fee-4995-b323-2532786c4044" containerName="console" containerID="cri-o://4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a" gracePeriod=15
Sep 17 16:45:42.641166 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.641140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:45:42.809278 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4lmm\" (UniqueName: \"kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809428 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809677 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config\") pod \"35a64c0c-4fee-4995-b323-2532786c4044\" (UID: \"35a64c0c-4fee-4995-b323-2532786c4044\") "
Sep 17 16:45:42.809677 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca" (OuterVolumeSpecName: "service-ca") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:42.809878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:42.810016 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:42.810016 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.809984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config" (OuterVolumeSpecName: "console-config") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:45:42.811749 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.811725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:42.811835 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.811754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:42.811835 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.811773    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm" (OuterVolumeSpecName: "kube-api-access-b4lmm") pod "35a64c0c-4fee-4995-b323-2532786c4044" (UID: "35a64c0c-4fee-4995-b323-2532786c4044"). InnerVolumeSpecName "kube-api-access-b4lmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:42.910017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.909995    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-oauth-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910017 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910015    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-console-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910140 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910024    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-service-ca\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910140 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910033    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4lmm\" (UniqueName: \"kubernetes.io/projected/35a64c0c-4fee-4995-b323-2532786c4044-kube-api-access-b4lmm\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910140 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910041    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35a64c0c-4fee-4995-b323-2532786c4044-console-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910140 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910050    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-trusted-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:42.910140 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:42.910059    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35a64c0c-4fee-4995-b323-2532786c4044-oauth-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:45:43.416301 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.416267    2580 generic.go:356] "Generic (PLEG): container finished" podID="35a64c0c-4fee-4995-b323-2532786c4044" containerID="4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a" exitCode=0
Sep 17 16:45:43.416663 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.416311    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764dc994d7-bcn42"
Sep 17 16:45:43.416663 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.416346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764dc994d7-bcn42" event={"ID":"35a64c0c-4fee-4995-b323-2532786c4044","Type":"ContainerDied","Data":"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"}
Sep 17 16:45:43.416663 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.416391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764dc994d7-bcn42" event={"ID":"35a64c0c-4fee-4995-b323-2532786c4044","Type":"ContainerDied","Data":"8b47c46021593b2482bb11eeff66c3176472c4dbf92c01e145dbefcb1f1eebd2"}
Sep 17 16:45:43.416663 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.416410    2580 scope.go:117] "RemoveContainer" containerID="4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"
Sep 17 16:45:43.424419 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.424402    2580 scope.go:117] "RemoveContainer" containerID="4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"
Sep 17 16:45:43.424663 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:45:43.424645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a\": container with ID starting with 4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a not found: ID does not exist" containerID="4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"
Sep 17 16:45:43.424712 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.424672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a"} err="failed to get container status \"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a\": rpc error: code = NotFound desc = could not find container \"4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a\": container with ID starting with 4141a7801ea58749a0597d38d11723d2561b552d5ea993c7236ec30cbd3bcc2a not found: ID does not exist"
Sep 17 16:45:43.431141 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.431120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:45:43.437628 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:43.437609    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-764dc994d7-bcn42"]
Sep 17 16:45:45.363565 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:45:45.363529    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a64c0c-4fee-4995-b323-2532786c4044" path="/var/lib/kubelet/pods/35a64c0c-4fee-4995-b323-2532786c4044/volumes"
Sep 17 16:46:49.289959 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:46:49.289932    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:46:49.294217 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:46:49.294195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:46:49.294725 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:46:49.294707    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:46:49.298712 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:46:49.298692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:46:49.301416 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:46:49.301400    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:49:00.901640 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.901561    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858899875d-56fmv"]
Sep 17 16:49:00.902141 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.901834    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a64c0c-4fee-4995-b323-2532786c4044" containerName="console"
Sep 17 16:49:00.902141 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.901852    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a64c0c-4fee-4995-b323-2532786c4044" containerName="console"
Sep 17 16:49:00.902141 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.901941    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a64c0c-4fee-4995-b323-2532786c4044" containerName="console"
Sep 17 16:49:00.904855 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.904834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.921858 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.921832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858899875d-56fmv"]
Sep 17 16:49:00.965913 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.965885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-trusted-ca-bundle\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966053 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.965935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-service-ca\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966053 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.965962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-oauth-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966137 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.966051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966137 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.966116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-oauth-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966137 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.966132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbh4q\" (UniqueName: \"kubernetes.io/projected/c91285f1-8c3d-407b-a6db-1f3d1e902db7-kube-api-access-nbh4q\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:00.966250 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:00.966152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066580 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-oauth-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066580 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbh4q\" (UniqueName: \"kubernetes.io/projected/c91285f1-8c3d-407b-a6db-1f3d1e902db7-kube-api-access-nbh4q\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-trusted-ca-bundle\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-service-ca\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-oauth-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.066778 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.066709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.067494 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.067468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-service-ca\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.067588 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.067504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.067588 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.067504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-oauth-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.067588 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.067570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c91285f1-8c3d-407b-a6db-1f3d1e902db7-trusted-ca-bundle\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.069563 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.069545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-oauth-config\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.069681 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.069664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c91285f1-8c3d-407b-a6db-1f3d1e902db7-console-serving-cert\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.076260 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.076242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbh4q\" (UniqueName: \"kubernetes.io/projected/c91285f1-8c3d-407b-a6db-1f3d1e902db7-kube-api-access-nbh4q\") pod \"console-858899875d-56fmv\" (UID: \"c91285f1-8c3d-407b-a6db-1f3d1e902db7\") " pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.215359 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.215289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:01.490148 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:01.490022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858899875d-56fmv"]
Sep 17 16:49:01.494216 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:49:01.494190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91285f1_8c3d_407b_a6db_1f3d1e902db7.slice/crio-e44c84b7e5abe9abe318cbc0356db2c7452aa38de03966be3fda8778811eb106 WatchSource:0}: Error finding container e44c84b7e5abe9abe318cbc0356db2c7452aa38de03966be3fda8778811eb106: Status 404 returned error can't find the container with id e44c84b7e5abe9abe318cbc0356db2c7452aa38de03966be3fda8778811eb106
Sep 17 16:49:02.139561 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:02.139525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858899875d-56fmv" event={"ID":"c91285f1-8c3d-407b-a6db-1f3d1e902db7","Type":"ContainerStarted","Data":"93ae0f5a41765a579bd1a7e2b7b0cf83306c8ded18bcf23bbb5a8b277cb48223"}
Sep 17 16:49:02.139952 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:02.139582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858899875d-56fmv" event={"ID":"c91285f1-8c3d-407b-a6db-1f3d1e902db7","Type":"ContainerStarted","Data":"e44c84b7e5abe9abe318cbc0356db2c7452aa38de03966be3fda8778811eb106"}
Sep 17 16:49:02.177398 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:02.177347    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858899875d-56fmv" podStartSLOduration=2.177331314 podStartE2EDuration="2.177331314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:49:02.175090922 +0000 UTC m=+434.872593120" watchObservedRunningTime="2026-09-17 16:49:02.177331314 +0000 UTC m=+434.874833578"
Sep 17 16:49:11.215926 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:11.215874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:11.216294 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:11.215968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:11.220559 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:11.220538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:12.179668 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:12.179640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858899875d-56fmv"
Sep 17 16:49:12.231305 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:12.231273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:49:37.251586 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.251523    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7568d66fcb-64vb9" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerName="console" containerID="cri-o://b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9" gracePeriod=15
Sep 17 16:49:37.325790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.325755    2580 patch_prober.go:28] interesting pod/console-7568d66fcb-64vb9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.22:8443/health\": dial tcp 10.133.0.22:8443: connect: connection refused" start-of-body=
Sep 17 16:49:37.325924 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.325808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7568d66fcb-64vb9" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerName="console" probeResult="failure" output="Get \"https://10.133.0.22:8443/health\": dial tcp 10.133.0.22:8443: connect: connection refused"
Sep 17 16:49:37.491345 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.491320    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:49:37.660520 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660542    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfw96\" (UniqueName: \"kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660696 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.660971 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.660737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca\") pod \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\" (UID: \"ff95c8fb-e35d-4f49-a8f1-61814b94cd68\") "
Sep 17 16:49:37.661025 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.661006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:37.661087 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.661039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:37.661137 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.661106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:37.661396 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.661318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config" (OuterVolumeSpecName: "console-config") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:49:37.662757 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.662733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96" (OuterVolumeSpecName: "kube-api-access-zfw96") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "kube-api-access-zfw96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:49:37.662757 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.662753    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:49:37.662901 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.662778    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff95c8fb-e35d-4f49-a8f1-61814b94cd68" (UID: "ff95c8fb-e35d-4f49-a8f1-61814b94cd68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:49:37.761551 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761524    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-trusted-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761551 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761548    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfw96\" (UniqueName: \"kubernetes.io/projected/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-kube-api-access-zfw96\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761562    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-oauth-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761574    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-oauth-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761586    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-serving-cert\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761598    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-console-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:37.761704 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:37.761609    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff95c8fb-e35d-4f49-a8f1-61814b94cd68-service-ca\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:49:38.272873 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.272836    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerID="b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9" exitCode=0
Sep 17 16:49:38.273254 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.272883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7568d66fcb-64vb9"
Sep 17 16:49:38.273254 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.272880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7568d66fcb-64vb9" event={"ID":"ff95c8fb-e35d-4f49-a8f1-61814b94cd68","Type":"ContainerDied","Data":"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"}
Sep 17 16:49:38.273254 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.273027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7568d66fcb-64vb9" event={"ID":"ff95c8fb-e35d-4f49-a8f1-61814b94cd68","Type":"ContainerDied","Data":"24e047e5afc69c07dcfe180ce1dbcb404d2501e71039a795d2a0080d00d91858"}
Sep 17 16:49:38.273254 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.273054    2580 scope.go:117] "RemoveContainer" containerID="b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"
Sep 17 16:49:38.281632 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.281615    2580 scope.go:117] "RemoveContainer" containerID="b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"
Sep 17 16:49:38.281924 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:49:38.281885    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9\": container with ID starting with b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9 not found: ID does not exist" containerID="b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"
Sep 17 16:49:38.281996 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.281927    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9"} err="failed to get container status \"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9\": rpc error: code = NotFound desc = could not find container \"b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9\": container with ID starting with b7203207c65b941c24cf66552a806a230d78013d5dc30a54fa1ea7ae8d78d2d9 not found: ID does not exist"
Sep 17 16:49:38.293804 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.293780    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:49:38.297269 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:38.297248    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7568d66fcb-64vb9"]
Sep 17 16:49:39.362842 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:49:39.362811    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" path="/var/lib/kubelet/pods/ff95c8fb-e35d-4f49-a8f1-61814b94cd68/volumes"
Sep 17 16:51:49.313474 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:51:49.313441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:51:49.315842 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:51:49.315820    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:51:49.319008 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:51:49.318984    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:51:49.320635 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:51:49.320616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:52:00.775315 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.775231    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:52:00.775853 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.775534    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerName="console"
Sep 17 16:52:00.775853 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.775547    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerName="console"
Sep 17 16:52:00.775853 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.775599    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff95c8fb-e35d-4f49-a8f1-61814b94cd68" containerName="console"
Sep 17 16:52:00.778476 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.778458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:00.781627 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.781606    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-77dxf\""
Sep 17 16:52:00.781748 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.781684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:52:00.781896 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.781876    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb976-kube-rbac-proxy-sar-config\""
Sep 17 16:52:00.782038 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.781876    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:52:00.782038 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.781936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb976-predictor-serving-cert\""
Sep 17 16:52:00.790799 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.790779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:52:00.929800 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.929751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdtf\" (UniqueName: \"kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:00.929987 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.929807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:00.929987 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:00.929887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.031336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.031259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdtf\" (UniqueName: \"kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.031336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.031295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.031336 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.031325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.031942 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.031894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.033783 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.033754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.041475 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.041454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdtf\" (UniqueName: \"kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf\") pod \"error-404-isvc-eb976-predictor-d669575b4-j4rjg\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") " pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.088547 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.088518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:01.363876 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.363848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:52:01.367188 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:52:01.367168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b95ad_0041_4199_b82b_2f245cc6f1e8.slice/crio-945e4912fda9195e53f4b4de2f7289482f949e76e3a68d94ae644477cc0acd17 WatchSource:0}: Error finding container 945e4912fda9195e53f4b4de2f7289482f949e76e3a68d94ae644477cc0acd17: Status 404 returned error can't find the container with id 945e4912fda9195e53f4b4de2f7289482f949e76e3a68d94ae644477cc0acd17
Sep 17 16:52:01.368837 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.368821    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:52:01.805420 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:01.805383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerStarted","Data":"945e4912fda9195e53f4b4de2f7289482f949e76e3a68d94ae644477cc0acd17"}
Sep 17 16:52:12.859158 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:12.859125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerStarted","Data":"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"}
Sep 17 16:52:15.874333 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:15.874292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerStarted","Data":"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"}
Sep 17 16:52:15.874708 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:15.874359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:15.874708 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:15.874523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:15.875812 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:15.875787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:52:15.892579 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:15.892537    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podStartSLOduration=2.392180753 podStartE2EDuration="15.892524539s" totalImagesPullingTime="13.500343786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:15.891100229 +0000 UTC m=+628.588602429" watchObservedRunningTime="2026-09-17 16:52:15.892524539 +0000 UTC m=+628.590026736"
Sep 17 16:52:16.878019 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:16.877977    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:52:21.882873 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:21.882844    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:52:21.883358 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:21.883308    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:52:31.883987 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:31.883944    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:52:41.883700 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:41.883654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:52:51.883991 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:52:51.883948    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:53:01.883870 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:01.883834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:53:20.187663 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.187632    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:53:20.191005 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.190990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.192940 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.192899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eb976-serving-cert\""
Sep 17 16:53:20.193059 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.193023    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eb976-kube-rbac-proxy-sar-config\""
Sep 17 16:53:20.203482 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.203459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:53:20.263289 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.263255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.263435 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.263313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.364640 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.364605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.364831 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.364658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.364831 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:20.364755    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-eb976-serving-cert: secret "switch-graph-eb976-serving-cert" not found
Sep 17 16:53:20.365000 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:20.364834    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls podName:1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6 nodeName:}" failed. No retries permitted until 2026-09-17 16:53:20.864817549 +0000 UTC m=+693.562319729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls") pod "switch-graph-eb976-84f6d675cc-txp68" (UID: "1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6") : secret "switch-graph-eb976-serving-cert" not found
Sep 17 16:53:20.365381 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.365359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.868942 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.868895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:20.871369 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:20.871340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") pod \"switch-graph-eb976-84f6d675cc-txp68\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") " pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:21.101223 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:21.101191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:21.374111 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:21.374090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:53:21.376548 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:53:21.376516    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce7023c_c51f_4b1d_9c6f_0c0f7d9f51f6.slice/crio-56cd7127ec88fa040060a207e53d205a5ca150eb2cea1cc25975b37a44dc7b48 WatchSource:0}: Error finding container 56cd7127ec88fa040060a207e53d205a5ca150eb2cea1cc25975b37a44dc7b48: Status 404 returned error can't find the container with id 56cd7127ec88fa040060a207e53d205a5ca150eb2cea1cc25975b37a44dc7b48
Sep 17 16:53:22.117643 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:22.117613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" event={"ID":"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6","Type":"ContainerStarted","Data":"56cd7127ec88fa040060a207e53d205a5ca150eb2cea1cc25975b37a44dc7b48"}
Sep 17 16:53:24.127348 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:24.127322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" event={"ID":"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6","Type":"ContainerStarted","Data":"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"}
Sep 17 16:53:24.127692 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:24.127446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:24.149124 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:24.149083    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podStartSLOduration=1.458808927 podStartE2EDuration="4.149067344s" totalImagesPullingTime="2.690258417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:24.147750157 +0000 UTC m=+696.845252357" watchObservedRunningTime="2026-09-17 16:53:24.149067344 +0000 UTC m=+696.846569545"
Sep 17 16:53:30.136371 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.136287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:30.366289 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.366257    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:53:30.366446 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.366400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" containerID="cri-o://2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4" gracePeriod=30
Sep 17 16:53:30.547749 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.547713    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:53:30.547992 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.547948    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" containerID="cri-o://3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe" gracePeriod=30
Sep 17 16:53:30.548139 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.547983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kube-rbac-proxy" containerID="cri-o://46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7" gracePeriod=30
Sep 17 16:53:30.953786 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.953687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 16:53:30.957748 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.957731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:30.961089 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.961067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5187-predictor-serving-cert\""
Sep 17 16:53:30.961343 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.961326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5187-kube-rbac-proxy-sar-config\""
Sep 17 16:53:30.983278 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:30.983257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 16:53:31.056472 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.056437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxgj\" (UniqueName: \"kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.056612 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.056484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.056612 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.056519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.157003 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.156973    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerID="46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7" exitCode=2
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.156973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxgj\" (UniqueName: \"kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.157048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerDied","Data":"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"}
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.157070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.157097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:31.157231    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b5187-predictor-serving-cert: secret "error-404-isvc-b5187-predictor-serving-cert" not found
Sep 17 16:53:31.157451 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:31.157298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls podName:71dbf396-3708-4a3e-af40-aaa9e222d4a3 nodeName:}" failed. No retries permitted until 2026-09-17 16:53:31.657278087 +0000 UTC m=+704.354780264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls") pod "error-404-isvc-b5187-predictor-d99857bc6-97v66" (UID: "71dbf396-3708-4a3e-af40-aaa9e222d4a3") : secret "error-404-isvc-b5187-predictor-serving-cert" not found
Sep 17 16:53:31.157854 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.157834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.172921 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.172884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxgj\" (UniqueName: \"kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.660665 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.660625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.663188 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.663159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") pod \"error-404-isvc-b5187-predictor-d99857bc6-97v66\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") " pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.868574 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.868544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:31.878892 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.878831    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 16:53:31.884248 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:31.884209    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 16:53:32.146670 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:32.146642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 16:53:32.149521 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:53:32.149496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dbf396_3708_4a3e_af40_aaa9e222d4a3.slice/crio-2858495c880eaf2b4808491d1291b8bb2c688d919bb9b76bacc8c17b9e302bbe WatchSource:0}: Error finding container 2858495c880eaf2b4808491d1291b8bb2c688d919bb9b76bacc8c17b9e302bbe: Status 404 returned error can't find the container with id 2858495c880eaf2b4808491d1291b8bb2c688d919bb9b76bacc8c17b9e302bbe
Sep 17 16:53:32.162094 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:32.162069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerStarted","Data":"2858495c880eaf2b4808491d1291b8bb2c688d919bb9b76bacc8c17b9e302bbe"}
Sep 17 16:53:33.168116 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.168083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerStarted","Data":"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"}
Sep 17 16:53:33.168614 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.168146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerStarted","Data":"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"}
Sep 17 16:53:33.168614 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.168282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:33.168614 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.168316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:33.169605 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.169580    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:53:33.195787 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.195740    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podStartSLOduration=3.195725489 podStartE2EDuration="3.195725489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:33.194458413 +0000 UTC m=+705.891960612" watchObservedRunningTime="2026-09-17 16:53:33.195725489 +0000 UTC m=+705.893227687"
Sep 17 16:53:33.488344 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.488325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:53:33.578432 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.578404    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqdtf\" (UniqueName: \"kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf\") pod \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") "
Sep 17 16:53:33.578432 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.578432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config\") pod \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") "
Sep 17 16:53:33.578619 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.578513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls\") pod \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\" (UID: \"9e6b95ad-0041-4199-b82b-2f245cc6f1e8\") "
Sep 17 16:53:33.578839 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.578820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e6b95ad-0041-4199-b82b-2f245cc6f1e8" (UID: "9e6b95ad-0041-4199-b82b-2f245cc6f1e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:53:33.580790 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.580772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e6b95ad-0041-4199-b82b-2f245cc6f1e8" (UID: "9e6b95ad-0041-4199-b82b-2f245cc6f1e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:53:33.580860 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.580792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf" (OuterVolumeSpecName: "kube-api-access-bqdtf") pod "9e6b95ad-0041-4199-b82b-2f245cc6f1e8" (UID: "9e6b95ad-0041-4199-b82b-2f245cc6f1e8"). InnerVolumeSpecName "kube-api-access-bqdtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:33.679177 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.679145    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:53:33.679177 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.679173    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdtf\" (UniqueName: \"kubernetes.io/projected/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-api-access-bqdtf\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:53:33.679346 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:33.679186    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e6b95ad-0041-4199-b82b-2f245cc6f1e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:53:34.173734 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.173698    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerID="3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe" exitCode=0
Sep 17 16:53:34.173734 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.173733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerDied","Data":"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"}
Sep 17 16:53:34.174252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.173752    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"
Sep 17 16:53:34.174252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.173797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg" event={"ID":"9e6b95ad-0041-4199-b82b-2f245cc6f1e8","Type":"ContainerDied","Data":"945e4912fda9195e53f4b4de2f7289482f949e76e3a68d94ae644477cc0acd17"}
Sep 17 16:53:34.174252 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.173825    2580 scope.go:117] "RemoveContainer" containerID="46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"
Sep 17 16:53:34.174512 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.174486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:53:34.182217 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.182199    2580 scope.go:117] "RemoveContainer" containerID="3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"
Sep 17 16:53:34.189232 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.189216    2580 scope.go:117] "RemoveContainer" containerID="46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"
Sep 17 16:53:34.189488 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:34.189470    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7\": container with ID starting with 46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7 not found: ID does not exist" containerID="46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"
Sep 17 16:53:34.189532 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.189497    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7"} err="failed to get container status \"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7\": rpc error: code = NotFound desc = could not find container \"46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7\": container with ID starting with 46ada8136b525c64ecaee9973db496eef32aa56f6bc65f0ff69faefb6aed9ee7 not found: ID does not exist"
Sep 17 16:53:34.189532 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.189513    2580 scope.go:117] "RemoveContainer" containerID="3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"
Sep 17 16:53:34.189754 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:53:34.189736    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe\": container with ID starting with 3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe not found: ID does not exist" containerID="3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"
Sep 17 16:53:34.189799 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.189771    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe"} err="failed to get container status \"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe\": rpc error: code = NotFound desc = could not find container \"3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe\": container with ID starting with 3917d1ce29b07928d18d32d77595c12d828fea907f2930e7e96d6f1164a4d2fe not found: ID does not exist"
Sep 17 16:53:34.193823 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.193802    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:53:34.198372 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:34.198354    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb976-predictor-d669575b4-j4rjg"]
Sep 17 16:53:35.134880 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:35.134844    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:35.363297 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:35.363265    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" path="/var/lib/kubelet/pods/9e6b95ad-0041-4199-b82b-2f245cc6f1e8/volumes"
Sep 17 16:53:39.179772 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:39.179741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:53:39.180284 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:39.180260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:53:40.135173 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:40.135137    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:45.135323 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:45.135282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:45.135777 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:45.135399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:53:49.180581 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:49.180538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:53:50.135162 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:50.135118    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:55.134376 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:55.134335    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:59.181240 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:53:59.181200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:54:00.135485 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:00.135447    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:00.997766 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:00.997735    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:54:01.104514 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.104483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") pod \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") "
Sep 17 16:54:01.104667 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.104559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle\") pod \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\" (UID: \"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6\") "
Sep 17 16:54:01.104927 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.104872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" (UID: "1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:01.106761 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.106735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" (UID: "1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:01.205451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.205380    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:54:01.205451 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.205403    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:54:01.278856 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.278822    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerID="2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4" exitCode=0
Sep 17 16:54:01.279039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.278865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"
Sep 17 16:54:01.279039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.278872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" event={"ID":"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6","Type":"ContainerDied","Data":"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"}
Sep 17 16:54:01.279039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.278946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68" event={"ID":"1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6","Type":"ContainerDied","Data":"56cd7127ec88fa040060a207e53d205a5ca150eb2cea1cc25975b37a44dc7b48"}
Sep 17 16:54:01.279039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.278977    2580 scope.go:117] "RemoveContainer" containerID="2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"
Sep 17 16:54:01.289208 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.289193    2580 scope.go:117] "RemoveContainer" containerID="2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"
Sep 17 16:54:01.289454 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:01.289432    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4\": container with ID starting with 2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4 not found: ID does not exist" containerID="2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"
Sep 17 16:54:01.289508 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.289463    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4"} err="failed to get container status \"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4\": rpc error: code = NotFound desc = could not find container \"2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4\": container with ID starting with 2d6a8bdc168f9613d3daf06537ba0b6f4181dace6af40867b099a97d250918f4 not found: ID does not exist"
Sep 17 16:54:01.298254 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.298234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:54:01.300126 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.300109    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eb976-84f6d675cc-txp68"]
Sep 17 16:54:01.362627 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:01.362599    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" path="/var/lib/kubelet/pods/1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6/volumes"
Sep 17 16:54:09.180238 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:09.180197    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 16:54:10.172881 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.172848    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:10.173211 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173194    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kube-rbac-proxy"
Sep 17 16:54:10.173298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173215    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kube-rbac-proxy"
Sep 17 16:54:10.173298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173237    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976"
Sep 17 16:54:10.173298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976"
Sep 17 16:54:10.173298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173273    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container"
Sep 17 16:54:10.173298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container"
Sep 17 16:54:10.173545 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173356    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kserve-container"
Sep 17 16:54:10.173545 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173374    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e6b95ad-0041-4199-b82b-2f245cc6f1e8" containerName="kube-rbac-proxy"
Sep 17 16:54:10.173545 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.173384    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce7023c-c51f-4b1d-9c6f-0c0f7d9f51f6" containerName="switch-graph-eb976"
Sep 17 16:54:10.177108 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.177081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.180397 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.180365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 16:54:10.180755 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.180369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 16:54:10.185205 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.185178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:10.275498 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.275472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.275622 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.275514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.376364 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.376339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.376481 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.376395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.376560 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:10.376541    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Sep 17 16:54:10.376626 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:10.376610    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls podName:d21cb18d-40c7-4662-9878-78a04ea77220 nodeName:}" failed. No retries permitted until 2026-09-17 16:54:10.876590245 +0000 UTC m=+743.574092425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls") pod "model-chainer-5f85d95997-z2skz" (UID: "d21cb18d-40c7-4662-9878-78a04ea77220") : secret "model-chainer-serving-cert" not found
Sep 17 16:54:10.377107 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.377089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.880237 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.880202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:10.882716 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:10.882684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") pod \"model-chainer-5f85d95997-z2skz\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") " pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:11.090992 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:11.090949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:11.363934 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:11.363893    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:11.364388 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:54:11.364366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21cb18d_40c7_4662_9878_78a04ea77220.slice/crio-00adba28ae5d8f3ec91ff2b0f9396994834efbdbd37075412bd6ba7b276a079d WatchSource:0}: Error finding container 00adba28ae5d8f3ec91ff2b0f9396994834efbdbd37075412bd6ba7b276a079d: Status 404 returned error can't find the container with id 00adba28ae5d8f3ec91ff2b0f9396994834efbdbd37075412bd6ba7b276a079d
Sep 17 16:54:12.323137 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:12.323101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" event={"ID":"d21cb18d-40c7-4662-9878-78a04ea77220","Type":"ContainerStarted","Data":"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"}
Sep 17 16:54:12.323317 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:12.323161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" event={"ID":"d21cb18d-40c7-4662-9878-78a04ea77220","Type":"ContainerStarted","Data":"00adba28ae5d8f3ec91ff2b0f9396994834efbdbd37075412bd6ba7b276a079d"}
Sep 17 16:54:12.323317 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:12.323264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:12.338896 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:12.338845    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podStartSLOduration=2.338831592 podStartE2EDuration="2.338831592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:12.338651767 +0000 UTC m=+745.036153966" watchObservedRunningTime="2026-09-17 16:54:12.338831592 +0000 UTC m=+745.036333789"
Sep 17 16:54:18.332833 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:18.332808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:19.181054 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:19.181025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 16:54:20.675060 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:20.675026    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:20.675417 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:20.675268    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" containerID="cri-o://ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7" gracePeriod=30
Sep 17 16:54:21.104966 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.104932    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 16:54:21.108931 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.108894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.111950 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.111928    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f3946-predictor-serving-cert\""
Sep 17 16:54:21.112074 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.111958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f3946-kube-rbac-proxy-sar-config\""
Sep 17 16:54:21.120425 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.120403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 16:54:21.147251 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.147222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscs5\" (UniqueName: \"kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.147251 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.147253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.147430 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.147283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.248139 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.248110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscs5\" (UniqueName: \"kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.248295 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.248151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.248295 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.248197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.248442 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:21.248315    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f3946-predictor-serving-cert: secret "error-404-isvc-f3946-predictor-serving-cert" not found
Sep 17 16:54:21.248442 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:21.248378    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls podName:b8018f98-181a-478f-8256-c694f5390374 nodeName:}" failed. No retries permitted until 2026-09-17 16:54:21.748359521 +0000 UTC m=+754.445861703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls") pod "error-404-isvc-f3946-predictor-c448bf9d-q7xtf" (UID: "b8018f98-181a-478f-8256-c694f5390374") : secret "error-404-isvc-f3946-predictor-serving-cert" not found
Sep 17 16:54:21.248884 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.248861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.257554 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.257527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscs5\" (UniqueName: \"kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.751186 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.751153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:21.753682 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:21.753661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") pod \"error-404-isvc-f3946-predictor-c448bf9d-q7xtf\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") " pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:22.019974 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:22.019884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:22.317081 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:22.317052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 16:54:22.319737 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:54:22.319712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8018f98_181a_478f_8256_c694f5390374.slice/crio-9eb7089158dd520d700e6b5e97eb7f4d1fea685888ecba01bf73b85edf639931 WatchSource:0}: Error finding container 9eb7089158dd520d700e6b5e97eb7f4d1fea685888ecba01bf73b85edf639931: Status 404 returned error can't find the container with id 9eb7089158dd520d700e6b5e97eb7f4d1fea685888ecba01bf73b85edf639931
Sep 17 16:54:22.372099 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:22.372075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerStarted","Data":"9eb7089158dd520d700e6b5e97eb7f4d1fea685888ecba01bf73b85edf639931"}
Sep 17 16:54:23.332053 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.332007    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:23.376878 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.376845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerStarted","Data":"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"}
Sep 17 16:54:23.377063 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.376895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerStarted","Data":"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"}
Sep 17 16:54:23.377063 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.376935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:23.377063 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.376957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:23.378212 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.378188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:54:23.398544 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:23.398508    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podStartSLOduration=2.398493577 podStartE2EDuration="2.398493577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:23.397783235 +0000 UTC m=+756.095285433" watchObservedRunningTime="2026-09-17 16:54:23.398493577 +0000 UTC m=+756.095995834"
Sep 17 16:54:24.381155 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:24.381114    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:54:28.331767 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:28.331720    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:29.386098 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:29.386074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:54:29.386535 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:29.386506    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:54:30.618225 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.618195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 16:54:30.621593 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.621574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.623600 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.623582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b5187-serving-cert\""
Sep 17 16:54:30.623705 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.623613    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b5187-kube-rbac-proxy-sar-config\""
Sep 17 16:54:30.631160 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.631138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 16:54:30.725142 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.725116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.725303 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.725160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.825589 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.825559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.825770 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.825626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.826331 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.826304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.828181 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.828160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls\") pod \"switch-graph-b5187-58856df79d-g7x99\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") " pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:30.932836 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:30.932764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:31.203718 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:31.203637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 16:54:31.206188 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:54:31.206159    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf482f510_a3d2_4933_a409_113ca1648880.slice/crio-7c50923717a6c49dc01762e86eb79f508ea1359409273d4927f693d49d4a7ffe WatchSource:0}: Error finding container 7c50923717a6c49dc01762e86eb79f508ea1359409273d4927f693d49d4a7ffe: Status 404 returned error can't find the container with id 7c50923717a6c49dc01762e86eb79f508ea1359409273d4927f693d49d4a7ffe
Sep 17 16:54:31.410019 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:31.409984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" event={"ID":"f482f510-a3d2-4933-a409-113ca1648880","Type":"ContainerStarted","Data":"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"}
Sep 17 16:54:31.410200 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:31.410031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" event={"ID":"f482f510-a3d2-4933-a409-113ca1648880","Type":"ContainerStarted","Data":"7c50923717a6c49dc01762e86eb79f508ea1359409273d4927f693d49d4a7ffe"}
Sep 17 16:54:31.410200 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:31.410092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:31.427445 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:31.427401    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podStartSLOduration=1.427387172 podStartE2EDuration="1.427387172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:31.426449763 +0000 UTC m=+764.123951962" watchObservedRunningTime="2026-09-17 16:54:31.427387172 +0000 UTC m=+764.124889369"
Sep 17 16:54:33.331742 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:33.331704    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:33.332127 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:33.331800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:37.419399 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:37.419371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 16:54:38.331290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:38.331256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:39.387468 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:39.387430    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:54:43.333128 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:43.333089    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:48.331515 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:48.331478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:49.387039 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:49.386996    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:54:50.811682 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.811661    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:50.895193 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.895168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") pod \"d21cb18d-40c7-4662-9878-78a04ea77220\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") "
Sep 17 16:54:50.895354 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.895221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle\") pod \"d21cb18d-40c7-4662-9878-78a04ea77220\" (UID: \"d21cb18d-40c7-4662-9878-78a04ea77220\") "
Sep 17 16:54:50.895557 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.895535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d21cb18d-40c7-4662-9878-78a04ea77220" (UID: "d21cb18d-40c7-4662-9878-78a04ea77220"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:50.897346 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.897327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d21cb18d-40c7-4662-9878-78a04ea77220" (UID: "d21cb18d-40c7-4662-9878-78a04ea77220"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:50.996290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.996188    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d21cb18d-40c7-4662-9878-78a04ea77220-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:54:50.996290 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:50.996232    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21cb18d-40c7-4662-9878-78a04ea77220-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 16:54:51.488255 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.488214    2580 generic.go:356] "Generic (PLEG): container finished" podID="d21cb18d-40c7-4662-9878-78a04ea77220" containerID="ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7" exitCode=0
Sep 17 16:54:51.488255 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.488257    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"
Sep 17 16:54:51.488462 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.488274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" event={"ID":"d21cb18d-40c7-4662-9878-78a04ea77220","Type":"ContainerDied","Data":"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"}
Sep 17 16:54:51.488462 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.488328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz" event={"ID":"d21cb18d-40c7-4662-9878-78a04ea77220","Type":"ContainerDied","Data":"00adba28ae5d8f3ec91ff2b0f9396994834efbdbd37075412bd6ba7b276a079d"}
Sep 17 16:54:51.488462 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.488349    2580 scope.go:117] "RemoveContainer" containerID="ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"
Sep 17 16:54:51.496298 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.496280    2580 scope.go:117] "RemoveContainer" containerID="ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"
Sep 17 16:54:51.496558 ip-10-0-143-112 kubenswrapper[2580]: E0917 16:54:51.496539    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7\": container with ID starting with ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7 not found: ID does not exist" containerID="ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"
Sep 17 16:54:51.496622 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.496570    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7"} err="failed to get container status \"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7\": rpc error: code = NotFound desc = could not find container \"ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7\": container with ID starting with ce1687b652e9639820db664aefd0b07da122685940b46bb6aa665e6a64ce88d7 not found: ID does not exist"
Sep 17 16:54:51.504602 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.504580    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:51.508860 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:51.508840    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5f85d95997-z2skz"]
Sep 17 16:54:53.363447 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:53.363413    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" path="/var/lib/kubelet/pods/d21cb18d-40c7-4662-9878-78a04ea77220/volumes"
Sep 17 16:54:59.387176 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:54:59.387090    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 16:55:09.387817 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:09.387787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 16:55:20.940325 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.940274    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 16:55:20.940801 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.940645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer"
Sep 17 16:55:20.940801 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.940663    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer"
Sep 17 16:55:20.940801 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.940733    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21cb18d-40c7-4662-9878-78a04ea77220" containerName="model-chainer"
Sep 17 16:55:20.943695 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.943679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:20.945747 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.945726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f3946-serving-cert\""
Sep 17 16:55:20.945862 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.945746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f3946-kube-rbac-proxy-sar-config\""
Sep 17 16:55:20.953226 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:20.953200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 16:55:21.021608 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.021569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.021770 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.021622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.123048 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.123016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.123202 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.123056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.123639 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.123610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.125644 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.125628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls\") pod \"sequence-graph-f3946-5c566544b8-gg6fs\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") " pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.254590 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.254518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.523124 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.523049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 16:55:21.525797 ip-10-0-143-112 kubenswrapper[2580]: W0917 16:55:21.525769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5937be_a7ad_457f_98e1_a21de7a7aa73.slice/crio-3774a161adc05de887dd62ed0c0c789b3a08f38849e48be8c36cfc9522e27785 WatchSource:0}: Error finding container 3774a161adc05de887dd62ed0c0c789b3a08f38849e48be8c36cfc9522e27785: Status 404 returned error can't find the container with id 3774a161adc05de887dd62ed0c0c789b3a08f38849e48be8c36cfc9522e27785
Sep 17 16:55:21.611799 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.611761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" event={"ID":"bc5937be-a7ad-457f-98e1-a21de7a7aa73","Type":"ContainerStarted","Data":"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"}
Sep 17 16:55:21.611965 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.611814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" event={"ID":"bc5937be-a7ad-457f-98e1-a21de7a7aa73","Type":"ContainerStarted","Data":"3774a161adc05de887dd62ed0c0c789b3a08f38849e48be8c36cfc9522e27785"}
Sep 17 16:55:21.611965 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.611840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:55:21.627195 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:21.627147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podStartSLOduration=1.6271302730000001 podStartE2EDuration="1.627130273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:21.626873864 +0000 UTC m=+814.324376063" watchObservedRunningTime="2026-09-17 16:55:21.627130273 +0000 UTC m=+814.324632472"
Sep 17 16:55:27.621219 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:55:27.621192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 16:56:49.341410 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:56:49.341381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:56:49.341827 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:56:49.341389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 16:56:49.346240 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:56:49.346219    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 16:56:49.346440 ip-10-0-143-112 kubenswrapper[2580]: I0917 16:56:49.346422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:01:49.366038 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:01:49.366012    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:01:49.370931 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:01:49.367788    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:01:49.375737 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:01:49.375715    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:01:49.376469 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:01:49.376451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:02:45.254645 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.254611    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 17:02:45.257144 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.254952    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" containerID="cri-o://ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d" gracePeriod=30
Sep 17 17:02:45.417232 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.417197    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 17:02:45.417475 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.417448    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container" containerID="cri-o://c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280" gracePeriod=30
Sep 17 17:02:45.417556 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.417448    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kube-rbac-proxy" containerID="cri-o://f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131" gracePeriod=30
Sep 17 17:02:45.598715 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.598682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:02:45.602048 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.602033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.604416 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.604389    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93f48-kube-rbac-proxy-sar-config\""
Sep 17 17:02:45.604536 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.604392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93f48-predictor-serving-cert\""
Sep 17 17:02:45.620853 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.620829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:02:45.646422 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.646392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk5s\" (UniqueName: \"kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.646533 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.646446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.646533 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.646471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.747833 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.747804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk5s\" (UniqueName: \"kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.747991 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.747857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.747991 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.747883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.748095 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:02:45.748035    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-93f48-predictor-serving-cert: secret "error-404-isvc-93f48-predictor-serving-cert" not found
Sep 17 17:02:45.748140 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:02:45.748104    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls podName:2121444d-bd09-4dce-a822-a4c175a56390 nodeName:}" failed. No retries permitted until 2026-09-17 17:02:46.248082727 +0000 UTC m=+1258.945584910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls") pod "error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" (UID: "2121444d-bd09-4dce-a822-a4c175a56390") : secret "error-404-isvc-93f48-predictor-serving-cert" not found
Sep 17 17:02:45.748736 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.748717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:45.756488 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:45.756460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk5s\" (UniqueName: \"kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:46.252716 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.252677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:46.255257 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.255232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") pod \"error-404-isvc-93f48-predictor-7bb978bc98-f8kx7\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") " pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:46.308063 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.308032    2580 generic.go:356] "Generic (PLEG): container finished" podID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerID="f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131" exitCode=2
Sep 17 17:02:46.308186 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.308098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerDied","Data":"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"}
Sep 17 17:02:46.514003 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.513921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:46.787379 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:46.787353    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:02:46.790246 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:02:46.790214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2121444d_bd09_4dce_a822_a4c175a56390.slice/crio-b078e4e75e2e99a2ba6963461170802299acc0cd44076b38b204b87cb8205a20 WatchSource:0}: Error finding container b078e4e75e2e99a2ba6963461170802299acc0cd44076b38b204b87cb8205a20: Status 404 returned error can't find the container with id b078e4e75e2e99a2ba6963461170802299acc0cd44076b38b204b87cb8205a20
Sep 17 17:02:47.313751 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.313713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerStarted","Data":"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"}
Sep 17 17:02:47.314141 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.313779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerStarted","Data":"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"}
Sep 17 17:02:47.314141 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.313800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerStarted","Data":"b078e4e75e2e99a2ba6963461170802299acc0cd44076b38b204b87cb8205a20"}
Sep 17 17:02:47.314141 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.313825    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:47.332678 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.332637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podStartSLOduration=2.332620453 podStartE2EDuration="2.332620453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:47.332018294 +0000 UTC m=+1260.029520504" watchObservedRunningTime="2026-09-17 17:02:47.332620453 +0000 UTC m=+1260.030122653"
Sep 17 17:02:47.418502 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:47.418470    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:02:48.317449 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.317417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:48.318713 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.318682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:02:48.557080 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.557057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 17:02:48.673664 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.673563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") pod \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") "
Sep 17 17:02:48.673664 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.673614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config\") pod \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") "
Sep 17 17:02:48.673664 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.673642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txxgj\" (UniqueName: \"kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj\") pod \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\" (UID: \"71dbf396-3708-4a3e-af40-aaa9e222d4a3\") "
Sep 17 17:02:48.674001 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.673980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71dbf396-3708-4a3e-af40-aaa9e222d4a3" (UID: "71dbf396-3708-4a3e-af40-aaa9e222d4a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:48.675845 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.675816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71dbf396-3708-4a3e-af40-aaa9e222d4a3" (UID: "71dbf396-3708-4a3e-af40-aaa9e222d4a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:48.675845 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.675829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj" (OuterVolumeSpecName: "kube-api-access-txxgj") pod "71dbf396-3708-4a3e-af40-aaa9e222d4a3" (UID: "71dbf396-3708-4a3e-af40-aaa9e222d4a3"). InnerVolumeSpecName "kube-api-access-txxgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:48.775101 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.775070    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71dbf396-3708-4a3e-af40-aaa9e222d4a3-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:02:48.775101 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.775097    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:02:48.775101 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:48.775107    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxgj\" (UniqueName: \"kubernetes.io/projected/71dbf396-3708-4a3e-af40-aaa9e222d4a3-kube-api-access-txxgj\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:02:49.322945 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.322893    2580 generic.go:356] "Generic (PLEG): container finished" podID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerID="c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280" exitCode=0
Sep 17 17:02:49.323397 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.322943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerDied","Data":"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"}
Sep 17 17:02:49.323397 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.322962    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"
Sep 17 17:02:49.323397 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.323015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66" event={"ID":"71dbf396-3708-4a3e-af40-aaa9e222d4a3","Type":"ContainerDied","Data":"2858495c880eaf2b4808491d1291b8bb2c688d919bb9b76bacc8c17b9e302bbe"}
Sep 17 17:02:49.323397 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.323059    2580 scope.go:117] "RemoveContainer" containerID="f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"
Sep 17 17:02:49.323614 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.323584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:02:49.332166 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.332151    2580 scope.go:117] "RemoveContainer" containerID="c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"
Sep 17 17:02:49.339643 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.339624    2580 scope.go:117] "RemoveContainer" containerID="f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"
Sep 17 17:02:49.339892 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:02:49.339871    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131\": container with ID starting with f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131 not found: ID does not exist" containerID="f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"
Sep 17 17:02:49.339974 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.339925    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131"} err="failed to get container status \"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131\": rpc error: code = NotFound desc = could not find container \"f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131\": container with ID starting with f2d8eb09dcf40915a9a7ecd01d0824001b356b2a8fc97c8365c3d1ce711e4131 not found: ID does not exist"
Sep 17 17:02:49.339974 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.339943    2580 scope.go:117] "RemoveContainer" containerID="c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"
Sep 17 17:02:49.340176 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:02:49.340160    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280\": container with ID starting with c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280 not found: ID does not exist" containerID="c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"
Sep 17 17:02:49.340218 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.340181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280"} err="failed to get container status \"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280\": rpc error: code = NotFound desc = could not find container \"c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280\": container with ID starting with c106bddafa87adb301830fc71da90a3345f626236cfe6fdd3e1d217ccfd24280 not found: ID does not exist"
Sep 17 17:02:49.346815 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.346794    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 17:02:49.351080 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.351061    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5187-predictor-d99857bc6-97v66"]
Sep 17 17:02:49.369141 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:49.369118    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" path="/var/lib/kubelet/pods/71dbf396-3708-4a3e-af40-aaa9e222d4a3/volumes"
Sep 17 17:02:52.417867 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:52.417832    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:02:54.327774 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:54.327740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:02:54.328229 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:54.328201    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:02:57.418103 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:57.418066    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:02:57.418500 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:02:57.418161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 17:03:02.418401 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:02.418359    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:04.328201 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:04.328156    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:03:07.418560 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:07.418523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:12.418176 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:12.418135    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:14.328194 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:14.328153    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:03:15.397054 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.397033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 17:03:15.424081 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.424052    2580 generic.go:356] "Generic (PLEG): container finished" podID="f482f510-a3d2-4933-a409-113ca1648880" containerID="ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d" exitCode=0
Sep 17 17:03:15.424219 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.424092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"
Sep 17 17:03:15.424219 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.424144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" event={"ID":"f482f510-a3d2-4933-a409-113ca1648880","Type":"ContainerDied","Data":"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"}
Sep 17 17:03:15.424219 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.424207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99" event={"ID":"f482f510-a3d2-4933-a409-113ca1648880","Type":"ContainerDied","Data":"7c50923717a6c49dc01762e86eb79f508ea1359409273d4927f693d49d4a7ffe"}
Sep 17 17:03:15.424386 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.424239    2580 scope.go:117] "RemoveContainer" containerID="ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"
Sep 17 17:03:15.432459 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.432436    2580 scope.go:117] "RemoveContainer" containerID="ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"
Sep 17 17:03:15.432711 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:15.432692    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d\": container with ID starting with ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d not found: ID does not exist" containerID="ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"
Sep 17 17:03:15.432781 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.432725    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d"} err="failed to get container status \"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d\": rpc error: code = NotFound desc = could not find container \"ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d\": container with ID starting with ab2cf84bfe60cea827eb223aaa112448f0e4e212effe087c5feca89f183b164d not found: ID does not exist"
Sep 17 17:03:15.497166 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.497138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle\") pod \"f482f510-a3d2-4933-a409-113ca1648880\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") "
Sep 17 17:03:15.497300 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.497194    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls\") pod \"f482f510-a3d2-4933-a409-113ca1648880\" (UID: \"f482f510-a3d2-4933-a409-113ca1648880\") "
Sep 17 17:03:15.497494 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.497463    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f482f510-a3d2-4933-a409-113ca1648880" (UID: "f482f510-a3d2-4933-a409-113ca1648880"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:15.499427 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.499404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f482f510-a3d2-4933-a409-113ca1648880" (UID: "f482f510-a3d2-4933-a409-113ca1648880"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:15.598534 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.598510    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f482f510-a3d2-4933-a409-113ca1648880-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:15.598534 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.598533    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f482f510-a3d2-4933-a409-113ca1648880-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:15.745012 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.744984    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 17:03:15.748196 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:15.748174    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b5187-58856df79d-g7x99"]
Sep 17 17:03:17.362917 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:17.362876    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f482f510-a3d2-4933-a409-113ca1648880" path="/var/lib/kubelet/pods/f482f510-a3d2-4933-a409-113ca1648880/volumes"
Sep 17 17:03:24.328571 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:24.328531    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:03:34.329067 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:34.329029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:03:35.650696 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.650664    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 17:03:35.651069 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.650854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" containerID="cri-o://d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f" gracePeriod=30
Sep 17 17:03:35.802970 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.802936    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 17:03:35.803191 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.803167    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container" containerID="cri-o://431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e" gracePeriod=30
Sep 17 17:03:35.803286 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.803222    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kube-rbac-proxy" containerID="cri-o://065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673" gracePeriod=30
Sep 17 17:03:35.954493 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.954460    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:03:35.954957 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.954942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kube-rbac-proxy"
Sep 17 17:03:35.955024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.954961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kube-rbac-proxy"
Sep 17 17:03:35.955024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.954974    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187"
Sep 17 17:03:35.955024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.954984    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187"
Sep 17 17:03:35.955024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.955007    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container"
Sep 17 17:03:35.955024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.955016    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container"
Sep 17 17:03:35.955192 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.955099    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kserve-container"
Sep 17 17:03:35.955192 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.955111    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f482f510-a3d2-4933-a409-113ca1648880" containerName="switch-graph-b5187"
Sep 17 17:03:35.955192 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.955122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dbf396-3708-4a3e-af40-aaa9e222d4a3" containerName="kube-rbac-proxy"
Sep 17 17:03:35.964764 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.964730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:35.967829 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.967795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e2a5-kube-rbac-proxy-sar-config\""
Sep 17 17:03:35.968252 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.968231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e2a5-predictor-serving-cert\""
Sep 17 17:03:35.970242 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:35.970220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:03:36.055454 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.055423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.055634 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.055468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.055634 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.055520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd77b\" (UniqueName: \"kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.156779 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.156743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd77b\" (UniqueName: \"kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.156981 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.156807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.156981 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.156836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.157479 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.157458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.159537 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.159510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.165223 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.165187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd77b\" (UniqueName: \"kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b\") pod \"error-404-isvc-2e2a5-predictor-65f657f664-rbwl2\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") " pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.277148 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.277083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:36.509008 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.508975    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8018f98-181a-478f-8256-c694f5390374" containerID="065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673" exitCode=2
Sep 17 17:03:36.509165 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.509047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerDied","Data":"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"}
Sep 17 17:03:36.551076 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:36.551052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:03:36.553507 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:03:36.553484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode034966f_1914_4d3e_95ad_be58587547b4.slice/crio-c40fc9e0c0021a9ebd3eabba5c8c2ce1d6bfd61b07ea6314e8789c587b5f9cb8 WatchSource:0}: Error finding container c40fc9e0c0021a9ebd3eabba5c8c2ce1d6bfd61b07ea6314e8789c587b5f9cb8: Status 404 returned error can't find the container with id c40fc9e0c0021a9ebd3eabba5c8c2ce1d6bfd61b07ea6314e8789c587b5f9cb8
Sep 17 17:03:37.514948 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.514898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerStarted","Data":"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"}
Sep 17 17:03:37.515318 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.514976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerStarted","Data":"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"}
Sep 17 17:03:37.515318 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.515004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerStarted","Data":"c40fc9e0c0021a9ebd3eabba5c8c2ce1d6bfd61b07ea6314e8789c587b5f9cb8"}
Sep 17 17:03:37.515318 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.515074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:37.534336 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.534289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podStartSLOduration=2.534277173 podStartE2EDuration="2.534277173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:37.533402666 +0000 UTC m=+1310.230904863" watchObservedRunningTime="2026-09-17 17:03:37.534277173 +0000 UTC m=+1310.231779370"
Sep 17 17:03:37.620027 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:37.619994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:38.518660 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.518621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:38.520175 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.520145    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:03:38.941615 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.941586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 17:03:38.976139 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.976111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kscs5\" (UniqueName: \"kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5\") pod \"b8018f98-181a-478f-8256-c694f5390374\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") "
Sep 17 17:03:38.976286 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.976196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config\") pod \"b8018f98-181a-478f-8256-c694f5390374\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") "
Sep 17 17:03:38.976286 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.976235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") pod \"b8018f98-181a-478f-8256-c694f5390374\" (UID: \"b8018f98-181a-478f-8256-c694f5390374\") "
Sep 17 17:03:38.976631 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.976607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8018f98-181a-478f-8256-c694f5390374" (UID: "b8018f98-181a-478f-8256-c694f5390374"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:38.978355 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.978329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5" (OuterVolumeSpecName: "kube-api-access-kscs5") pod "b8018f98-181a-478f-8256-c694f5390374" (UID: "b8018f98-181a-478f-8256-c694f5390374"). InnerVolumeSpecName "kube-api-access-kscs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:38.978617 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:38.978604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8018f98-181a-478f-8256-c694f5390374" (UID: "b8018f98-181a-478f-8256-c694f5390374"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:39.076888 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.076816    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8018f98-181a-478f-8256-c694f5390374-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:39.076888 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.076843    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscs5\" (UniqueName: \"kubernetes.io/projected/b8018f98-181a-478f-8256-c694f5390374-kube-api-access-kscs5\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:39.076888 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.076855    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8018f98-181a-478f-8256-c694f5390374-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:39.523571 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.523528    2580 generic.go:356] "Generic (PLEG): container finished" podID="b8018f98-181a-478f-8256-c694f5390374" containerID="431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e" exitCode=0
Sep 17 17:03:39.524035 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.523580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"
Sep 17 17:03:39.524035 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.523610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerDied","Data":"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"}
Sep 17 17:03:39.524035 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.523666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf" event={"ID":"b8018f98-181a-478f-8256-c694f5390374","Type":"ContainerDied","Data":"9eb7089158dd520d700e6b5e97eb7f4d1fea685888ecba01bf73b85edf639931"}
Sep 17 17:03:39.524035 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.523694    2580 scope.go:117] "RemoveContainer" containerID="065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"
Sep 17 17:03:39.524304 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.524282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:03:39.532375 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.532354    2580 scope.go:117] "RemoveContainer" containerID="431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"
Sep 17 17:03:39.539379 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.539363    2580 scope.go:117] "RemoveContainer" containerID="065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"
Sep 17 17:03:39.539649 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:39.539631    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673\": container with ID starting with 065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673 not found: ID does not exist" containerID="065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"
Sep 17 17:03:39.539708 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.539656    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673"} err="failed to get container status \"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673\": rpc error: code = NotFound desc = could not find container \"065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673\": container with ID starting with 065d102a2bccff49f1b62c1d1c29cd3abc27265aab45f55a1356d83a4e212673 not found: ID does not exist"
Sep 17 17:03:39.539708 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.539671    2580 scope.go:117] "RemoveContainer" containerID="431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"
Sep 17 17:03:39.539970 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:39.539896    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e\": container with ID starting with 431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e not found: ID does not exist" containerID="431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"
Sep 17 17:03:39.540027 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.539975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e"} err="failed to get container status \"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e\": rpc error: code = NotFound desc = could not find container \"431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e\": container with ID starting with 431fca7ae5ff4b6862215ea3b576844af5e209f2acf913935a41d1dad188d19e not found: ID does not exist"
Sep 17 17:03:39.546772 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.546750    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 17:03:39.551405 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:39.551385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f3946-predictor-c448bf9d-q7xtf"]
Sep 17 17:03:41.362616 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:41.362581    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8018f98-181a-478f-8256-c694f5390374" path="/var/lib/kubelet/pods/b8018f98-181a-478f-8256-c694f5390374/volumes"
Sep 17 17:03:42.619703 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:42.619662    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:44.529052 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:44.529020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:03:44.529548 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:44.529385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:03:45.516227 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516191    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:03:45.516548 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516535    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kube-rbac-proxy"
Sep 17 17:03:45.516600 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516552    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kube-rbac-proxy"
Sep 17 17:03:45.516600 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516572    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container"
Sep 17 17:03:45.516600 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516578    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container"
Sep 17 17:03:45.516698 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516649    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kserve-container"
Sep 17 17:03:45.516698 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.516659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8018f98-181a-478f-8256-c694f5390374" containerName="kube-rbac-proxy"
Sep 17 17:03:45.520958 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.520943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.523126 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.523098    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-93f48-kube-rbac-proxy-sar-config\""
Sep 17 17:03:45.523279 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.523261    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-93f48-serving-cert\""
Sep 17 17:03:45.530301 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.530280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:03:45.629734 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.629706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.629734 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.629736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.731077 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.731051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.731077 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.731078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.731644 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.731623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.733636 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.733612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls\") pod \"ensemble-graph-93f48-78c9465b4b-nx7ln\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") " pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:45.832217 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:45.832191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:46.105503 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:46.105331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:03:46.108377 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:03:46.108346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f45309_d518_4eca_af9d_af9a678fc39e.slice/crio-33bd9191689d1d3446ed3286164c19e1e1cf090b2097f3406cf747e8c124aafa WatchSource:0}: Error finding container 33bd9191689d1d3446ed3286164c19e1e1cf090b2097f3406cf747e8c124aafa: Status 404 returned error can't find the container with id 33bd9191689d1d3446ed3286164c19e1e1cf090b2097f3406cf747e8c124aafa
Sep 17 17:03:46.552020 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:46.551983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" event={"ID":"b0f45309-d518-4eca-af9d-af9a678fc39e","Type":"ContainerStarted","Data":"89397349a45343ebe4550517d90b65b64e2c708dbadbd73318161c1be779e1ef"}
Sep 17 17:03:46.552461 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:46.552046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" event={"ID":"b0f45309-d518-4eca-af9d-af9a678fc39e","Type":"ContainerStarted","Data":"33bd9191689d1d3446ed3286164c19e1e1cf090b2097f3406cf747e8c124aafa"}
Sep 17 17:03:46.552461 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:46.552073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:46.569028 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:46.568988    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podStartSLOduration=1.568973483 podStartE2EDuration="1.568973483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:46.567771025 +0000 UTC m=+1319.265273223" watchObservedRunningTime="2026-09-17 17:03:46.568973483 +0000 UTC m=+1319.266475677"
Sep 17 17:03:47.619252 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:47.619217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:47.619652 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:47.619312    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 17:03:52.562490 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:52.562463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:03:52.620320 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:52.620282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:54.529853 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:54.529804    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:03:55.682574 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:55.682543    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:03:55.682958 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:55.682731    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" containerID="cri-o://89397349a45343ebe4550517d90b65b64e2c708dbadbd73318161c1be779e1ef" gracePeriod=30
Sep 17 17:03:55.994632 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:55.994551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:03:55.994813 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:55.994792    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container" containerID="cri-o://4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63" gracePeriod=30
Sep 17 17:03:55.994954 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:55.994929    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kube-rbac-proxy" containerID="cri-o://db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d" gracePeriod=30
Sep 17 17:03:56.351804 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.351772    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:03:56.355037 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.355021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.358191 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.358171    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d52cc-kube-rbac-proxy-sar-config\""
Sep 17 17:03:56.358302 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.358244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d52cc-predictor-serving-cert\""
Sep 17 17:03:56.374119 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.374099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:03:56.422389 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.422355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.422389 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.422387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98mj\" (UniqueName: \"kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.422561 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.422435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.523479 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.523441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98mj\" (UniqueName: \"kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.523633 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.523500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.523633 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.523567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.523633 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:56.523611    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-serving-cert: secret "error-404-isvc-d52cc-predictor-serving-cert" not found
Sep 17 17:03:56.523787 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:56.523677    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls podName:12984265-2b03-4c3e-a875-877b439e2d94 nodeName:}" failed. No retries permitted until 2026-09-17 17:03:57.02365713 +0000 UTC m=+1329.721159321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls") pod "error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" (UID: "12984265-2b03-4c3e-a875-877b439e2d94") : secret "error-404-isvc-d52cc-predictor-serving-cert" not found
Sep 17 17:03:56.524210 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.524192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.533266 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.533248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98mj\" (UniqueName: \"kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:56.594976 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.594946    2580 generic.go:356] "Generic (PLEG): container finished" podID="2121444d-bd09-4dce-a822-a4c175a56390" containerID="db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d" exitCode=2
Sep 17 17:03:56.595103 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:56.595036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerDied","Data":"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"}
Sep 17 17:03:57.029103 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.029067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:57.031654 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.031629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") pod \"error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") " pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:57.264703 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.264667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:57.542581 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.542557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:03:57.543673 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:03:57.543650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12984265_2b03_4c3e_a875_877b439e2d94.slice/crio-106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05 WatchSource:0}: Error finding container 106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05: Status 404 returned error can't find the container with id 106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05
Sep 17 17:03:57.560443 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.560378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:57.601657 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.601628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerStarted","Data":"106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05"}
Sep 17 17:03:57.620108 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:57.620082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:03:58.607581 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:58.607544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerStarted","Data":"b1b9ed4ea4736f69c44b04568f1d38d1f900942d8ea1062217604f02de7b6be6"}
Sep 17 17:03:58.607974 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:58.607613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerStarted","Data":"cb4463121ed297fa564867bfd3e803acc4352206d0eb3d9ba454e9fdd03fab10"}
Sep 17 17:03:58.607974 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:58.607680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:58.645448 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:58.645398    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podStartSLOduration=2.645383854 podStartE2EDuration="2.645383854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:58.645089602 +0000 UTC m=+1331.342591802" watchObservedRunningTime="2026-09-17 17:03:58.645383854 +0000 UTC m=+1331.342886052"
Sep 17 17:03:59.132697 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.132674    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:03:59.250763 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.250684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfk5s\" (UniqueName: \"kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s\") pod \"2121444d-bd09-4dce-a822-a4c175a56390\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") "
Sep 17 17:03:59.250763 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.250729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config\") pod \"2121444d-bd09-4dce-a822-a4c175a56390\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") "
Sep 17 17:03:59.250972 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.250788    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") pod \"2121444d-bd09-4dce-a822-a4c175a56390\" (UID: \"2121444d-bd09-4dce-a822-a4c175a56390\") "
Sep 17 17:03:59.251187 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.251154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2121444d-bd09-4dce-a822-a4c175a56390" (UID: "2121444d-bd09-4dce-a822-a4c175a56390"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:59.252959 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.252936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2121444d-bd09-4dce-a822-a4c175a56390" (UID: "2121444d-bd09-4dce-a822-a4c175a56390"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:59.253027 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.253004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s" (OuterVolumeSpecName: "kube-api-access-pfk5s") pod "2121444d-bd09-4dce-a822-a4c175a56390" (UID: "2121444d-bd09-4dce-a822-a4c175a56390"). InnerVolumeSpecName "kube-api-access-pfk5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:59.352134 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.352098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2121444d-bd09-4dce-a822-a4c175a56390-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:59.352134 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.352128    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2121444d-bd09-4dce-a822-a4c175a56390-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:59.352134 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.352139    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfk5s\" (UniqueName: \"kubernetes.io/projected/2121444d-bd09-4dce-a822-a4c175a56390-kube-api-access-pfk5s\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:03:59.613519 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.613488    2580 generic.go:356] "Generic (PLEG): container finished" podID="2121444d-bd09-4dce-a822-a4c175a56390" containerID="4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63" exitCode=0
Sep 17 17:03:59.613896 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.613535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"
Sep 17 17:03:59.613896 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.613571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerDied","Data":"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"}
Sep 17 17:03:59.613896 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.613632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7" event={"ID":"2121444d-bd09-4dce-a822-a4c175a56390","Type":"ContainerDied","Data":"b078e4e75e2e99a2ba6963461170802299acc0cd44076b38b204b87cb8205a20"}
Sep 17 17:03:59.613896 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.613659    2580 scope.go:117] "RemoveContainer" containerID="db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"
Sep 17 17:03:59.614606 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.614064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:03:59.615659 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.615634    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:03:59.622211 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.622189    2580 scope.go:117] "RemoveContainer" containerID="4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"
Sep 17 17:03:59.630359 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.630336    2580 scope.go:117] "RemoveContainer" containerID="db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"
Sep 17 17:03:59.630685 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:59.630659    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d\": container with ID starting with db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d not found: ID does not exist" containerID="db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"
Sep 17 17:03:59.630795 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.630697    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d"} err="failed to get container status \"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d\": rpc error: code = NotFound desc = could not find container \"db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d\": container with ID starting with db133636a190f1aae52907b09f2b9d17aee219eaae0af58813f4ce59b698256d not found: ID does not exist"
Sep 17 17:03:59.630795 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.630720    2580 scope.go:117] "RemoveContainer" containerID="4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"
Sep 17 17:03:59.630795 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.630770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:03:59.630996 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:03:59.630972    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63\": container with ID starting with 4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63 not found: ID does not exist" containerID="4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"
Sep 17 17:03:59.631051 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.630992    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63"} err="failed to get container status \"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63\": rpc error: code = NotFound desc = could not find container \"4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63\": container with ID starting with 4c72a9cda504ad719f93fdf211d266b88ae4a4dfcceb64bc060b43d5a9ad3f63 not found: ID does not exist"
Sep 17 17:03:59.638122 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:03:59.638102    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93f48-predictor-7bb978bc98-f8kx7"]
Sep 17 17:04:00.618181 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:00.618141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:04:01.363453 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:01.363419    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2121444d-bd09-4dce-a822-a4c175a56390" path="/var/lib/kubelet/pods/2121444d-bd09-4dce-a822-a4c175a56390/volumes"
Sep 17 17:04:02.560833 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:02.560785    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:02.619549 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:02.619520    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:04.529635 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:04.529575    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:04:05.623538 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.623511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:04:05.624132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.623978    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:04:05.796341 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.796321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 17:04:05.906055 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.905986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle\") pod \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") "
Sep 17 17:04:05.906055 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.906050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls\") pod \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\" (UID: \"bc5937be-a7ad-457f-98e1-a21de7a7aa73\") "
Sep 17 17:04:05.906413 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.906385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bc5937be-a7ad-457f-98e1-a21de7a7aa73" (UID: "bc5937be-a7ad-457f-98e1-a21de7a7aa73"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:05.908246 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:05.908230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc5937be-a7ad-457f-98e1-a21de7a7aa73" (UID: "bc5937be-a7ad-457f-98e1-a21de7a7aa73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:06.007495 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.007462    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc5937be-a7ad-457f-98e1-a21de7a7aa73-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:06.007495 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.007491    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5937be-a7ad-457f-98e1-a21de7a7aa73-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:06.643777 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.643746    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerID="d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f" exitCode=137
Sep 17 17:04:06.644176 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.643788    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"
Sep 17 17:04:06.644176 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.643806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" event={"ID":"bc5937be-a7ad-457f-98e1-a21de7a7aa73","Type":"ContainerDied","Data":"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"}
Sep 17 17:04:06.644176 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.643847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs" event={"ID":"bc5937be-a7ad-457f-98e1-a21de7a7aa73","Type":"ContainerDied","Data":"3774a161adc05de887dd62ed0c0c789b3a08f38849e48be8c36cfc9522e27785"}
Sep 17 17:04:06.644176 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.643870    2580 scope.go:117] "RemoveContainer" containerID="d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"
Sep 17 17:04:06.652484 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.652462    2580 scope.go:117] "RemoveContainer" containerID="d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"
Sep 17 17:04:06.652752 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:04:06.652733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f\": container with ID starting with d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f not found: ID does not exist" containerID="d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"
Sep 17 17:04:06.652809 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.652760    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f"} err="failed to get container status \"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f\": rpc error: code = NotFound desc = could not find container \"d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f\": container with ID starting with d941f252b5a350c2d79bd4692c60bd902472f73db8dbdee9646bc9fb9b133a7f not found: ID does not exist"
Sep 17 17:04:06.666023 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.665998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 17:04:06.669630 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:06.669609    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f3946-5c566544b8-gg6fs"]
Sep 17 17:04:07.363203 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:07.363173    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" path="/var/lib/kubelet/pods/bc5937be-a7ad-457f-98e1-a21de7a7aa73/volumes"
Sep 17 17:04:07.560671 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:07.560636    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:07.560832 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:07.560732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:04:12.560513 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:12.560469    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:14.529700 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:14.529659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 17:04:15.624582 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:15.624545    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:04:17.561011 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:17.560976    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:22.560142 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:22.560106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:24.530106 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:24.530072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:04:25.625046 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.625005    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:04:25.723926 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.723867    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerID="89397349a45343ebe4550517d90b65b64e2c708dbadbd73318161c1be779e1ef" exitCode=0
Sep 17 17:04:25.724078 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.723939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" event={"ID":"b0f45309-d518-4eca-af9d-af9a678fc39e","Type":"ContainerDied","Data":"89397349a45343ebe4550517d90b65b64e2c708dbadbd73318161c1be779e1ef"}
Sep 17 17:04:25.821296 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.821275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:04:25.964804 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.964723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls\") pod \"b0f45309-d518-4eca-af9d-af9a678fc39e\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") "
Sep 17 17:04:25.964804 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.964803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle\") pod \"b0f45309-d518-4eca-af9d-af9a678fc39e\" (UID: \"b0f45309-d518-4eca-af9d-af9a678fc39e\") "
Sep 17 17:04:25.965169 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.965147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b0f45309-d518-4eca-af9d-af9a678fc39e" (UID: "b0f45309-d518-4eca-af9d-af9a678fc39e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:25.966933 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:25.966893    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0f45309-d518-4eca-af9d-af9a678fc39e" (UID: "b0f45309-d518-4eca-af9d-af9a678fc39e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:26.065636 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.065610    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f45309-d518-4eca-af9d-af9a678fc39e-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:26.065636 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.065634    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0f45309-d518-4eca-af9d-af9a678fc39e-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:26.729962 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.729930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"
Sep 17 17:04:26.729962 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.729945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln" event={"ID":"b0f45309-d518-4eca-af9d-af9a678fc39e","Type":"ContainerDied","Data":"33bd9191689d1d3446ed3286164c19e1e1cf090b2097f3406cf747e8c124aafa"}
Sep 17 17:04:26.730411 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.730006    2580 scope.go:117] "RemoveContainer" containerID="89397349a45343ebe4550517d90b65b64e2c708dbadbd73318161c1be779e1ef"
Sep 17 17:04:26.752389 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.752365    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:04:26.755378 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:26.755360    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-93f48-78c9465b4b-nx7ln"]
Sep 17 17:04:27.363065 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:27.363033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" path="/var/lib/kubelet/pods/b0f45309-d518-4eca-af9d-af9a678fc39e/volumes"
Sep 17 17:04:35.624178 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:35.624141    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 17:04:36.061331 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061301    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:04:36.061592 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061579    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946"
Sep 17 17:04:36.061637 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061594    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946"
Sep 17 17:04:36.061637 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48"
Sep 17 17:04:36.061637 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48"
Sep 17 17:04:36.061637 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061626    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kube-rbac-proxy"
Sep 17 17:04:36.061637 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061631    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kube-rbac-proxy"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061644    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061696    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f45309-d518-4eca-af9d-af9a678fc39e" containerName="ensemble-graph-93f48"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061705    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kserve-container"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5937be-a7ad-457f-98e1-a21de7a7aa73" containerName="sequence-graph-f3946"
Sep 17 17:04:36.061792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.061719    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2121444d-bd09-4dce-a822-a4c175a56390" containerName="kube-rbac-proxy"
Sep 17 17:04:36.064384 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.064369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:04:36.064461 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.064452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.066692 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.066673    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2e2a5-kube-rbac-proxy-sar-config\""
Sep 17 17:04:36.066805 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.066698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2e2a5-serving-cert\""
Sep 17 17:04:36.244024 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.243990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.244164 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.244051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.344651 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.344570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.344651 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.344622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.345257 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.345231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.347112 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.347092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls\") pod \"sequence-graph-2e2a5-848d4d594c-z5css\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") " pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.374151 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.374127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.646942 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.646853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:04:36.650359 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:04:36.650318    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a3bc83_0538_4cfa_8643_d8f9a4263aca.slice/crio-15b7a96776f56411ec86d22cbfe2e1c53fce1ec8c76731b83741b4d9f2ddc59e WatchSource:0}: Error finding container 15b7a96776f56411ec86d22cbfe2e1c53fce1ec8c76731b83741b4d9f2ddc59e: Status 404 returned error can't find the container with id 15b7a96776f56411ec86d22cbfe2e1c53fce1ec8c76731b83741b4d9f2ddc59e
Sep 17 17:04:36.770418 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.770380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" event={"ID":"f1a3bc83-0538-4cfa-8643-d8f9a4263aca","Type":"ContainerStarted","Data":"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"}
Sep 17 17:04:36.770657 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.770436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" event={"ID":"f1a3bc83-0538-4cfa-8643-d8f9a4263aca","Type":"ContainerStarted","Data":"15b7a96776f56411ec86d22cbfe2e1c53fce1ec8c76731b83741b4d9f2ddc59e"}
Sep 17 17:04:36.770657 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.770482    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:36.791583 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:36.791544    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podStartSLOduration=1.791529143 podStartE2EDuration="1.791529143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:36.790489852 +0000 UTC m=+1369.487992050" watchObservedRunningTime="2026-09-17 17:04:36.791529143 +0000 UTC m=+1369.489031391"
Sep 17 17:04:42.780034 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:42.780009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:04:45.625087 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:45.625054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:04:45.943303 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:45.943226    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:04:45.943433 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:45.943400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" containerID="cri-o://ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea" gracePeriod=30
Sep 17 17:04:46.087891 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.087850    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:04:46.088152 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.088128    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container" containerID="cri-o://24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e" gracePeriod=30
Sep 17 17:04:46.088244 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.088215    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kube-rbac-proxy" containerID="cri-o://e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f" gracePeriod=30
Sep 17 17:04:46.290446 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.290418    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:04:46.293685 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.293668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.296154 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.296132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c2858-predictor-serving-cert\""
Sep 17 17:04:46.296276 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.296203    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c2858-kube-rbac-proxy-sar-config\""
Sep 17 17:04:46.304614 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.304593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:04:46.320924 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.320879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.321028 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.320936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbbd\" (UniqueName: \"kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.321028 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.320998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.422278 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.422248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.422400 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.422288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbbd\" (UniqueName: \"kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.422400 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.422328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.422515 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:04:46.422479    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c2858-predictor-serving-cert: secret "error-404-isvc-c2858-predictor-serving-cert" not found
Sep 17 17:04:46.422568 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:04:46.422546    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls podName:3fc6eedb-eff1-40e8-bb93-27bc190ca2ff nodeName:}" failed. No retries permitted until 2026-09-17 17:04:46.92252599 +0000 UTC m=+1379.620028183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls") pod "error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" (UID: "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff") : secret "error-404-isvc-c2858-predictor-serving-cert" not found
Sep 17 17:04:46.422967 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.422948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.430650 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.430626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbbd\" (UniqueName: \"kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.809950 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.809889    2580 generic.go:356] "Generic (PLEG): container finished" podID="e034966f-1914-4d3e-95ad-be58587547b4" containerID="e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f" exitCode=2
Sep 17 17:04:46.810388 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.809970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerDied","Data":"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"}
Sep 17 17:04:46.926674 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.926643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:46.929193 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:46.929161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") pod \"error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") " pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:47.203894 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.203804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:47.481131 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.481107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:04:47.483605 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:04:47.483577    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc6eedb_eff1_40e8_bb93_27bc190ca2ff.slice/crio-74d8555e7133c5252bd463e6b00ed3c8bf04ef4a0f94950034f4915fc9e0312a WatchSource:0}: Error finding container 74d8555e7133c5252bd463e6b00ed3c8bf04ef4a0f94950034f4915fc9e0312a: Status 404 returned error can't find the container with id 74d8555e7133c5252bd463e6b00ed3c8bf04ef4a0f94950034f4915fc9e0312a
Sep 17 17:04:47.779438 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.779349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:47.817205 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.817171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerStarted","Data":"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"}
Sep 17 17:04:47.817610 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.817247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerStarted","Data":"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"}
Sep 17 17:04:47.817610 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.817267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerStarted","Data":"74d8555e7133c5252bd463e6b00ed3c8bf04ef4a0f94950034f4915fc9e0312a"}
Sep 17 17:04:47.817610 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.817290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:47.817610 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.817309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:47.818555 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.818530    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:04:47.845303 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:47.845251    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podStartSLOduration=1.845236324 podStartE2EDuration="1.845236324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:47.843686856 +0000 UTC m=+1380.541189055" watchObservedRunningTime="2026-09-17 17:04:47.845236324 +0000 UTC m=+1380.542738522"
Sep 17 17:04:48.822433 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:48.822388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:04:49.318826 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.318804    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:04:49.342884 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.342855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls\") pod \"e034966f-1914-4d3e-95ad-be58587547b4\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") "
Sep 17 17:04:49.343029 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.342964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd77b\" (UniqueName: \"kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b\") pod \"e034966f-1914-4d3e-95ad-be58587547b4\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") "
Sep 17 17:04:49.343029 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.343019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config\") pod \"e034966f-1914-4d3e-95ad-be58587547b4\" (UID: \"e034966f-1914-4d3e-95ad-be58587547b4\") "
Sep 17 17:04:49.343363 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.343332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e034966f-1914-4d3e-95ad-be58587547b4" (UID: "e034966f-1914-4d3e-95ad-be58587547b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:04:49.345115 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.345090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e034966f-1914-4d3e-95ad-be58587547b4" (UID: "e034966f-1914-4d3e-95ad-be58587547b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:49.345194 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.345088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b" (OuterVolumeSpecName: "kube-api-access-zd77b") pod "e034966f-1914-4d3e-95ad-be58587547b4" (UID: "e034966f-1914-4d3e-95ad-be58587547b4"). InnerVolumeSpecName "kube-api-access-zd77b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:04:49.444000 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.443933    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e034966f-1914-4d3e-95ad-be58587547b4-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:49.444000 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.443954    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd77b\" (UniqueName: \"kubernetes.io/projected/e034966f-1914-4d3e-95ad-be58587547b4-kube-api-access-zd77b\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:49.444000 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.443966    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e034966f-1914-4d3e-95ad-be58587547b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:04:49.828408 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.828377    2580 generic.go:356] "Generic (PLEG): container finished" podID="e034966f-1914-4d3e-95ad-be58587547b4" containerID="24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e" exitCode=0
Sep 17 17:04:49.828859 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.828490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"
Sep 17 17:04:49.828859 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.828515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerDied","Data":"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"}
Sep 17 17:04:49.828859 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.828582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2" event={"ID":"e034966f-1914-4d3e-95ad-be58587547b4","Type":"ContainerDied","Data":"c40fc9e0c0021a9ebd3eabba5c8c2ce1d6bfd61b07ea6314e8789c587b5f9cb8"}
Sep 17 17:04:49.828859 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.828612    2580 scope.go:117] "RemoveContainer" containerID="e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"
Sep 17 17:04:49.836862 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.836842    2580 scope.go:117] "RemoveContainer" containerID="24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"
Sep 17 17:04:49.844244 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.844215    2580 scope.go:117] "RemoveContainer" containerID="e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"
Sep 17 17:04:49.844307 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.844276    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:04:49.844500 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:04:49.844484    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f\": container with ID starting with e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f not found: ID does not exist" containerID="e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"
Sep 17 17:04:49.844542 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.844511    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f"} err="failed to get container status \"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f\": rpc error: code = NotFound desc = could not find container \"e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f\": container with ID starting with e79ee7e8ac840a3601232e0ba21f1c0b36a6677112a48fe2a6efba6d6fe0374f not found: ID does not exist"
Sep 17 17:04:49.844542 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.844528    2580 scope.go:117] "RemoveContainer" containerID="24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"
Sep 17 17:04:49.844795 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:04:49.844778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e\": container with ID starting with 24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e not found: ID does not exist" containerID="24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"
Sep 17 17:04:49.844845 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.844799    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e"} err="failed to get container status \"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e\": rpc error: code = NotFound desc = could not find container \"24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e\": container with ID starting with 24b1ed1898dededf871af24f43fa618962b5fe3117b6e782279e36447859dc9e not found: ID does not exist"
Sep 17 17:04:49.848414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:49.848395    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e2a5-predictor-65f657f664-rbwl2"]
Sep 17 17:04:51.362803 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:51.362767    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e034966f-1914-4d3e-95ad-be58587547b4" path="/var/lib/kubelet/pods/e034966f-1914-4d3e-95ad-be58587547b4/volumes"
Sep 17 17:04:52.778783 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:52.778746    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:53.828288 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:53.828255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:04:53.828774 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:53.828749    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:04:55.944043 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944007    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:04:55.944414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944316    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kube-rbac-proxy"
Sep 17 17:04:55.944414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944328    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kube-rbac-proxy"
Sep 17 17:04:55.944414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944339    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container"
Sep 17 17:04:55.944414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944344    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container"
Sep 17 17:04:55.944414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944413    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kserve-container"
Sep 17 17:04:55.944594 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.944425    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e034966f-1914-4d3e-95ad-be58587547b4" containerName="kube-rbac-proxy"
Sep 17 17:04:55.948813 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.948797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:55.951066 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.951047    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d52cc-serving-cert\""
Sep 17 17:04:55.951189 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.951065    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d52cc-kube-rbac-proxy-sar-config\""
Sep 17 17:04:55.955805 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.955783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:04:55.983243 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.983212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:55.983358 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:55.983260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.084121 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.084095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.084121 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.084124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.084687 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.084663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.086586 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.086564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls\") pod \"ensemble-graph-d52cc-6b7fc7b8b9-nfzxq\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") " pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.260558 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.260480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.532936 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.532752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:04:56.536466 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:04:56.536439    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29005d12_e2aa_4f36_bbb0_b8db3d767cb3.slice/crio-5463837831f8348dec26f7d89f76a34f4b67f793bb592b7d5f92bad62ca01556 WatchSource:0}: Error finding container 5463837831f8348dec26f7d89f76a34f4b67f793bb592b7d5f92bad62ca01556: Status 404 returned error can't find the container with id 5463837831f8348dec26f7d89f76a34f4b67f793bb592b7d5f92bad62ca01556
Sep 17 17:04:56.863438 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.863342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" event={"ID":"29005d12-e2aa-4f36-bbb0-b8db3d767cb3","Type":"ContainerStarted","Data":"98985850631a51af80319f8487f3dc64c0a688bce891515323b7955aa1f066c8"}
Sep 17 17:04:56.863438 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.863397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" event={"ID":"29005d12-e2aa-4f36-bbb0-b8db3d767cb3","Type":"ContainerStarted","Data":"5463837831f8348dec26f7d89f76a34f4b67f793bb592b7d5f92bad62ca01556"}
Sep 17 17:04:56.863438 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.863430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:04:56.880061 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:56.879993    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podStartSLOduration=1.87997705 podStartE2EDuration="1.87997705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:56.879641973 +0000 UTC m=+1389.577144172" watchObservedRunningTime="2026-09-17 17:04:56.87997705 +0000 UTC m=+1389.577479251"
Sep 17 17:04:57.779135 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:57.779095    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:04:57.779519 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:04:57.779207    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:05:02.778456 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:02.778422    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:05:02.872898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:02.872868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:05:03.829108 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:03.829069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:05:07.778501 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:07.778467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:05:12.778813 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:12.778771    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:05:13.829720 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:13.829681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:05:16.082872 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.082847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:05:16.135737 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.135713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle\") pod \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") "
Sep 17 17:05:16.135875 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.135753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls\") pod \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\" (UID: \"f1a3bc83-0538-4cfa-8643-d8f9a4263aca\") "
Sep 17 17:05:16.136059 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.136037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f1a3bc83-0538-4cfa-8643-d8f9a4263aca" (UID: "f1a3bc83-0538-4cfa-8643-d8f9a4263aca"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:05:16.137839 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.137818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1a3bc83-0538-4cfa-8643-d8f9a4263aca" (UID: "f1a3bc83-0538-4cfa-8643-d8f9a4263aca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:05:16.236880 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.236822    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:05:16.236880 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.236844    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a3bc83-0538-4cfa-8643-d8f9a4263aca-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:05:16.948571 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.948538    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerID="ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea" exitCode=0
Sep 17 17:05:16.948735 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.948582    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"
Sep 17 17:05:16.948735 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.948584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" event={"ID":"f1a3bc83-0538-4cfa-8643-d8f9a4263aca","Type":"ContainerDied","Data":"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"}
Sep 17 17:05:16.948735 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.948634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css" event={"ID":"f1a3bc83-0538-4cfa-8643-d8f9a4263aca","Type":"ContainerDied","Data":"15b7a96776f56411ec86d22cbfe2e1c53fce1ec8c76731b83741b4d9f2ddc59e"}
Sep 17 17:05:16.948735 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.948666    2580 scope.go:117] "RemoveContainer" containerID="ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"
Sep 17 17:05:16.958091 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.958073    2580 scope.go:117] "RemoveContainer" containerID="ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"
Sep 17 17:05:16.958349 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:05:16.958331    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea\": container with ID starting with ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea not found: ID does not exist" containerID="ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"
Sep 17 17:05:16.958421 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.958357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea"} err="failed to get container status \"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea\": rpc error: code = NotFound desc = could not find container \"ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea\": container with ID starting with ea0bb4d7e70b5fa6ba024d5b89ea8317f7e97c7f65266acce490e43ba3a232ea not found: ID does not exist"
Sep 17 17:05:16.969242 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.969217    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:05:16.972811 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:16.972713    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2e2a5-848d4d594c-z5css"]
Sep 17 17:05:17.363110 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:17.363080    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" path="/var/lib/kubelet/pods/f1a3bc83-0538-4cfa-8643-d8f9a4263aca/volumes"
Sep 17 17:05:23.829164 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:23.829092    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:05:33.830105 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:33.830028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:05:46.221597 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.221564    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:05:46.222058 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.221859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5"
Sep 17 17:05:46.222058 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.221870    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5"
Sep 17 17:05:46.222058 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.221953    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a3bc83-0538-4cfa-8643-d8f9a4263aca" containerName="sequence-graph-2e2a5"
Sep 17 17:05:46.224876 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.224853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.227017 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.226998    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c2858-serving-cert\""
Sep 17 17:05:46.227111 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.227034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c2858-kube-rbac-proxy-sar-config\""
Sep 17 17:05:46.233706 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.233674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:05:46.365042 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.365019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.365183 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.365047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.466204 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.466176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.466204 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.466207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.466405 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:05:46.466327    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-c2858-serving-cert: secret "sequence-graph-c2858-serving-cert" not found
Sep 17 17:05:46.466405 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:05:46.466383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls podName:e7e38d44-028f-4511-a80d-9132da1aaa6c nodeName:}" failed. No retries permitted until 2026-09-17 17:05:46.966369499 +0000 UTC m=+1439.663871675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls") pod "sequence-graph-c2858-54876bd879-lh79j" (UID: "e7e38d44-028f-4511-a80d-9132da1aaa6c") : secret "sequence-graph-c2858-serving-cert" not found
Sep 17 17:05:46.466846 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.466823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.970677 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.970641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:46.973128 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:46.973111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") pod \"sequence-graph-c2858-54876bd879-lh79j\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") " pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:47.135559 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:47.135524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:47.412054 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:47.411978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:05:47.414615 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:05:47.414584    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e38d44_028f_4511_a80d_9132da1aaa6c.slice/crio-2407183efe3ba802c02ee8c9a9e84cec0b6579c7ad3b1224b475a2f0fd3f1e91 WatchSource:0}: Error finding container 2407183efe3ba802c02ee8c9a9e84cec0b6579c7ad3b1224b475a2f0fd3f1e91: Status 404 returned error can't find the container with id 2407183efe3ba802c02ee8c9a9e84cec0b6579c7ad3b1224b475a2f0fd3f1e91
Sep 17 17:05:48.070434 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:48.070396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" event={"ID":"e7e38d44-028f-4511-a80d-9132da1aaa6c","Type":"ContainerStarted","Data":"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"}
Sep 17 17:05:48.070621 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:48.070453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" event={"ID":"e7e38d44-028f-4511-a80d-9132da1aaa6c","Type":"ContainerStarted","Data":"2407183efe3ba802c02ee8c9a9e84cec0b6579c7ad3b1224b475a2f0fd3f1e91"}
Sep 17 17:05:48.070621 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:48.070498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:05:48.087630 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:48.087579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podStartSLOduration=2.087565557 podStartE2EDuration="2.087565557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:05:48.086137286 +0000 UTC m=+1440.783639485" watchObservedRunningTime="2026-09-17 17:05:48.087565557 +0000 UTC m=+1440.785067754"
Sep 17 17:05:54.081003 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:05:54.080974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:06:49.394583 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:06:49.394553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:06:49.397157 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:06:49.397134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:06:49.399316 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:06:49.399292    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:06:49.401947 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:06:49.401931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:11:49.421305 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:11:49.421276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:11:49.424884 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:11:49.424856    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:11:49.426444 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:11:49.426421    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:11:49.429603 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:11:49.429587    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:13:10.647031 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.646935    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:13:10.729032 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.728981    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" containerID="cri-o://98985850631a51af80319f8487f3dc64c0a688bce891515323b7955aa1f066c8" gracePeriod=30
Sep 17 17:13:10.799065 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.799033    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:13:10.799400 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.799301    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container" containerID="cri-o://b1b9ed4ea4736f69c44b04568f1d38d1f900942d8ea1062217604f02de7b6be6" gracePeriod=30
Sep 17 17:13:10.799400 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.799360    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kube-rbac-proxy" containerID="cri-o://cb4463121ed297fa564867bfd3e803acc4352206d0eb3d9ba454e9fdd03fab10" gracePeriod=30
Sep 17 17:13:10.917728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.917701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:13:10.920857 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.920837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:10.923330 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.923314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2fdf6-predictor-serving-cert\""
Sep 17 17:13:10.923425 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.923350    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2fdf6-kube-rbac-proxy-sar-config\""
Sep 17 17:13:10.934377 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.934354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:13:10.990658 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.990630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:10.990765 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.990704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rll\" (UniqueName: \"kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:10.990765 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:10.990750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.091466 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.091433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.091612 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.091488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rll\" (UniqueName: \"kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.091612 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.091520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.092224 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.092153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.094134 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.094111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.099348 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.099328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rll\" (UniqueName: \"kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll\") pod \"error-404-isvc-2fdf6-predictor-fff5f778b-5tld6\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") " pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.231478 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.231407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.507462 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.507397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:13:11.511173 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:13:11.511140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a66aa8_3b74_4249_84ab_669884df195b.slice/crio-20e746d35185496e771a73544c6588ff9790a548a85d089ffbb8d554c4a3edd2 WatchSource:0}: Error finding container 20e746d35185496e771a73544c6588ff9790a548a85d089ffbb8d554c4a3edd2: Status 404 returned error can't find the container with id 20e746d35185496e771a73544c6588ff9790a548a85d089ffbb8d554c4a3edd2
Sep 17 17:13:11.734876 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.734845    2580 generic.go:356] "Generic (PLEG): container finished" podID="12984265-2b03-4c3e-a875-877b439e2d94" containerID="cb4463121ed297fa564867bfd3e803acc4352206d0eb3d9ba454e9fdd03fab10" exitCode=2
Sep 17 17:13:11.735300 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.734938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerDied","Data":"cb4463121ed297fa564867bfd3e803acc4352206d0eb3d9ba454e9fdd03fab10"}
Sep 17 17:13:11.736744 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.736720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerStarted","Data":"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"}
Sep 17 17:13:11.736852 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.736771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerStarted","Data":"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"}
Sep 17 17:13:11.736852 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.736790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerStarted","Data":"20e746d35185496e771a73544c6588ff9790a548a85d089ffbb8d554c4a3edd2"}
Sep 17 17:13:11.736955 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.736932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.737001 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.736990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:11.738339 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.738314    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:11.757339 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:11.757272    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podStartSLOduration=1.7572594559999999 podStartE2EDuration="1.757259456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:11.756252431 +0000 UTC m=+1884.453754643" watchObservedRunningTime="2026-09-17 17:13:11.757259456 +0000 UTC m=+1884.454761654"
Sep 17 17:13:12.740649 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:12.740604    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:12.872115 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:12.872075    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:13.746378 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.746344    2580 generic.go:356] "Generic (PLEG): container finished" podID="12984265-2b03-4c3e-a875-877b439e2d94" containerID="b1b9ed4ea4736f69c44b04568f1d38d1f900942d8ea1062217604f02de7b6be6" exitCode=0
Sep 17 17:13:13.746708 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.746407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerDied","Data":"b1b9ed4ea4736f69c44b04568f1d38d1f900942d8ea1062217604f02de7b6be6"}
Sep 17 17:13:13.746708 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.746478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x" event={"ID":"12984265-2b03-4c3e-a875-877b439e2d94","Type":"ContainerDied","Data":"106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05"}
Sep 17 17:13:13.746708 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.746505    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106b38f5ed1a0d243a5ab46512c899830a6e2d315c7f239169abb90e4070fa05"
Sep 17 17:13:13.749881 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.749865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:13:13.810878 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.810852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config\") pod \"12984265-2b03-4c3e-a875-877b439e2d94\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") "
Sep 17 17:13:13.811079 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.810888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98mj\" (UniqueName: \"kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj\") pod \"12984265-2b03-4c3e-a875-877b439e2d94\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") "
Sep 17 17:13:13.811079 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.810957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") pod \"12984265-2b03-4c3e-a875-877b439e2d94\" (UID: \"12984265-2b03-4c3e-a875-877b439e2d94\") "
Sep 17 17:13:13.811203 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.811184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12984265-2b03-4c3e-a875-877b439e2d94" (UID: "12984265-2b03-4c3e-a875-877b439e2d94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:13:13.813114 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.813093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12984265-2b03-4c3e-a875-877b439e2d94" (UID: "12984265-2b03-4c3e-a875-877b439e2d94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:13:13.813197 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.813178    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj" (OuterVolumeSpecName: "kube-api-access-k98mj") pod "12984265-2b03-4c3e-a875-877b439e2d94" (UID: "12984265-2b03-4c3e-a875-877b439e2d94"). InnerVolumeSpecName "kube-api-access-k98mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:13:13.911501 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.911464    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12984265-2b03-4c3e-a875-877b439e2d94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:13:13.911501 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.911493    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98mj\" (UniqueName: \"kubernetes.io/projected/12984265-2b03-4c3e-a875-877b439e2d94-kube-api-access-k98mj\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:13:13.911501 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:13.911506    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12984265-2b03-4c3e-a875-877b439e2d94-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:13:14.749869 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:14.749831    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"
Sep 17 17:13:14.769880 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:14.769853    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:13:14.774135 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:14.774112    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d52cc-predictor-65bf9c9c88-b7k7x"]
Sep 17 17:13:15.363429 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:15.363389    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12984265-2b03-4c3e-a875-877b439e2d94" path="/var/lib/kubelet/pods/12984265-2b03-4c3e-a875-877b439e2d94/volumes"
Sep 17 17:13:17.745727 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:17.745692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:13:17.746433 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:17.746405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:17.871429 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:17.871394    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:22.872072 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:22.872036    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:22.872491 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:22.872153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:13:27.747135 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:27.747093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:27.871805 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:27.871766    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:32.871896 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:32.871853    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:37.747083 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:37.747046    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:37.872159 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:37.872124    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:13:40.856265 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:40.856231    2580 generic.go:356] "Generic (PLEG): container finished" podID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerID="98985850631a51af80319f8487f3dc64c0a688bce891515323b7955aa1f066c8" exitCode=0
Sep 17 17:13:40.856598 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:40.856309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" event={"ID":"29005d12-e2aa-4f36-bbb0-b8db3d767cb3","Type":"ContainerDied","Data":"98985850631a51af80319f8487f3dc64c0a688bce891515323b7955aa1f066c8"}
Sep 17 17:13:40.901095 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:40.901066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:13:41.034791 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.034767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle\") pod \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") "
Sep 17 17:13:41.034995 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.034828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls\") pod \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\" (UID: \"29005d12-e2aa-4f36-bbb0-b8db3d767cb3\") "
Sep 17 17:13:41.035223 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.035194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "29005d12-e2aa-4f36-bbb0-b8db3d767cb3" (UID: "29005d12-e2aa-4f36-bbb0-b8db3d767cb3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:13:41.037025 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.037003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29005d12-e2aa-4f36-bbb0-b8db3d767cb3" (UID: "29005d12-e2aa-4f36-bbb0-b8db3d767cb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:13:41.135546 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.135517    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:13:41.135546 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.135545    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29005d12-e2aa-4f36-bbb0-b8db3d767cb3-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:13:41.861376 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.861344    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"
Sep 17 17:13:41.861954 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.861389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq" event={"ID":"29005d12-e2aa-4f36-bbb0-b8db3d767cb3","Type":"ContainerDied","Data":"5463837831f8348dec26f7d89f76a34f4b67f793bb592b7d5f92bad62ca01556"}
Sep 17 17:13:41.861954 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.861448    2580 scope.go:117] "RemoveContainer" containerID="98985850631a51af80319f8487f3dc64c0a688bce891515323b7955aa1f066c8"
Sep 17 17:13:41.876181 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.876157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:13:41.879706 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:41.879681    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d52cc-6b7fc7b8b9-nfzxq"]
Sep 17 17:13:43.362790 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:43.362758    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" path="/var/lib/kubelet/pods/29005d12-e2aa-4f36-bbb0-b8db3d767cb3/volumes"
Sep 17 17:13:47.746842 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:47.746801    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:13:49.522019 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:49.521991    2580 scope.go:117] "RemoveContainer" containerID="cb4463121ed297fa564867bfd3e803acc4352206d0eb3d9ba454e9fdd03fab10"
Sep 17 17:13:49.529881 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:49.529865    2580 scope.go:117] "RemoveContainer" containerID="b1b9ed4ea4736f69c44b04568f1d38d1f900942d8ea1062217604f02de7b6be6"
Sep 17 17:13:57.747005 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:13:57.746975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:14:00.822026 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:00.821990    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:14:00.822487 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:00.822247    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" containerID="cri-o://7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4" gracePeriod=30
Sep 17 17:14:00.938755 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:00.938725    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:14:00.938976 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:00.938954    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" containerID="cri-o://16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd" gracePeriod=30
Sep 17 17:14:00.939058 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:00.938959    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kube-rbac-proxy" containerID="cri-o://02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3" gracePeriod=30
Sep 17 17:14:01.154521 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:14:01.154775 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc"
Sep 17 17:14:01.154841 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154776    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc"
Sep 17 17:14:01.154841 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kube-rbac-proxy"
Sep 17 17:14:01.154841 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154796    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kube-rbac-proxy"
Sep 17 17:14:01.154841 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154808    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container"
Sep 17 17:14:01.154841 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container"
Sep 17 17:14:01.155132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154862    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kserve-container"
Sep 17 17:14:01.155132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154871    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29005d12-e2aa-4f36-bbb0-b8db3d767cb3" containerName="ensemble-graph-d52cc"
Sep 17 17:14:01.155132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.154878    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12984265-2b03-4c3e-a875-877b439e2d94" containerName="kube-rbac-proxy"
Sep 17 17:14:01.157851 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.157834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.160149 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.160129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f0d22-predictor-serving-cert\""
Sep 17 17:14:01.160256 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.160145    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f0d22-kube-rbac-proxy-sar-config\""
Sep 17 17:14:01.169775 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.169751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:14:01.294802 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.294768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.295003 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.294841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.295003 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.294892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvws\" (UniqueName: \"kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.396153 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.396122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.396296 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.396167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvws\" (UniqueName: \"kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.396296 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.396211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.396425 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:01.396291    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-serving-cert: secret "error-404-isvc-f0d22-predictor-serving-cert" not found
Sep 17 17:14:01.396425 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:01.396371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls podName:e1621874-4f05-4d26-9144-22a9ad3db714 nodeName:}" failed. No retries permitted until 2026-09-17 17:14:01.896348387 +0000 UTC m=+1934.593850564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls") pod "error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" (UID: "e1621874-4f05-4d26-9144-22a9ad3db714") : secret "error-404-isvc-f0d22-predictor-serving-cert" not found
Sep 17 17:14:01.397031 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.397004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.405191 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.405148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvws\" (UniqueName: \"kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.900925 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.900870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.903370 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.903352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") pod \"error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") " pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:01.939451 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.939424    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerID="02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3" exitCode=2
Sep 17 17:14:01.939595 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:01.939469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerDied","Data":"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"}
Sep 17 17:14:02.068674 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.068640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:02.339191 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.339159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:14:02.342224 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:14:02.342200    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1621874_4f05_4d26_9144_22a9ad3db714.slice/crio-727d8ae96a5c250ba266e418ba58bd684038eb2a8d4c602c54f60294420f840e WatchSource:0}: Error finding container 727d8ae96a5c250ba266e418ba58bd684038eb2a8d4c602c54f60294420f840e: Status 404 returned error can't find the container with id 727d8ae96a5c250ba266e418ba58bd684038eb2a8d4c602c54f60294420f840e
Sep 17 17:14:02.944796 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.944763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerStarted","Data":"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"}
Sep 17 17:14:02.945200 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.944824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerStarted","Data":"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"}
Sep 17 17:14:02.945200 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.944850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerStarted","Data":"727d8ae96a5c250ba266e418ba58bd684038eb2a8d4c602c54f60294420f840e"}
Sep 17 17:14:02.945200 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.944875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:02.945200 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.944902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:02.946317 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.946289    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:02.964038 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:02.963997    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podStartSLOduration=1.9639833580000001 podStartE2EDuration="1.963983358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:02.963148117 +0000 UTC m=+1935.660650317" watchObservedRunningTime="2026-09-17 17:14:02.963983358 +0000 UTC m=+1935.661485556"
Sep 17 17:14:03.875542 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.875513    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:14:03.916630 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.916608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config\") pod \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") "
Sep 17 17:14:03.916764 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.916682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") pod \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") "
Sep 17 17:14:03.916764 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.916722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxbbd\" (UniqueName: \"kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd\") pod \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\" (UID: \"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff\") "
Sep 17 17:14:03.917062 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.917034    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" (UID: "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:03.918957 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.918936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd" (OuterVolumeSpecName: "kube-api-access-nxbbd") pod "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" (UID: "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff"). InnerVolumeSpecName "kube-api-access-nxbbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:03.919022 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.918962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" (UID: "3fc6eedb-eff1-40e8-bb93-27bc190ca2ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:03.949748 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.949717    2580 generic.go:356] "Generic (PLEG): container finished" podID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerID="16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd" exitCode=0
Sep 17 17:14:03.950091 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.949773    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"
Sep 17 17:14:03.950091 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.949802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerDied","Data":"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"}
Sep 17 17:14:03.950091 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.949860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" event={"ID":"3fc6eedb-eff1-40e8-bb93-27bc190ca2ff","Type":"ContainerDied","Data":"74d8555e7133c5252bd463e6b00ed3c8bf04ef4a0f94950034f4915fc9e0312a"}
Sep 17 17:14:03.950091 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.949886    2580 scope.go:117] "RemoveContainer" containerID="02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"
Sep 17 17:14:03.950431 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.950408    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:03.958403 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.958390    2580 scope.go:117] "RemoveContainer" containerID="16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"
Sep 17 17:14:03.965618 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.965604    2580 scope.go:117] "RemoveContainer" containerID="02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"
Sep 17 17:14:03.965881 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:03.965862    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3\": container with ID starting with 02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3 not found: ID does not exist" containerID="02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"
Sep 17 17:14:03.966009 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.965893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3"} err="failed to get container status \"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3\": rpc error: code = NotFound desc = could not find container \"02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3\": container with ID starting with 02fd953236429d25d14eb461932f6162ce514eda7f60c34040663be2df8b42b3 not found: ID does not exist"
Sep 17 17:14:03.966106 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.966016    2580 scope.go:117] "RemoveContainer" containerID="16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"
Sep 17 17:14:03.966280 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:03.966262    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd\": container with ID starting with 16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd not found: ID does not exist" containerID="16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"
Sep 17 17:14:03.966330 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.966286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd"} err="failed to get container status \"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd\": rpc error: code = NotFound desc = could not find container \"16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd\": container with ID starting with 16d6458902d3a156892d6aceb9120cfeb29275da56e27ed91dab3610f9c366dd not found: ID does not exist"
Sep 17 17:14:03.969358 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.969339    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:14:03.976125 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:03.976107    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9"]
Sep 17 17:14:04.017275 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.017211    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.017367 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.017343    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbbd\" (UniqueName: \"kubernetes.io/projected/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-api-access-nxbbd\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.017412 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.017366    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:04.079711 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.079687    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:04.823437 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.823385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:14:04.829468 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:04.829425    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c2858-predictor-69ddfb9f7d-bfxf9" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: i/o timeout"
Sep 17 17:14:05.367043 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:05.367002    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" path="/var/lib/kubelet/pods/3fc6eedb-eff1-40e8-bb93-27bc190ca2ff/volumes"
Sep 17 17:14:08.954792 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:08.954760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:08.955251 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:08.955224    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:09.079340 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:09.079300    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:10.858835 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.858805    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:10.859235 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859163    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container"
Sep 17 17:14:10.859235 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859176    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container"
Sep 17 17:14:10.859235 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859185    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kube-rbac-proxy"
Sep 17 17:14:10.859235 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859191    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kube-rbac-proxy"
Sep 17 17:14:10.859235 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859237    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kube-rbac-proxy"
Sep 17 17:14:10.859415 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.859248    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc6eedb-eff1-40e8-bb93-27bc190ca2ff" containerName="kserve-container"
Sep 17 17:14:10.862099 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.862081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.864390 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.864369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2fdf6-kube-rbac-proxy-sar-config\""
Sep 17 17:14:10.864507 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.864447    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2fdf6-serving-cert\""
Sep 17 17:14:10.866162 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.866140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.866250 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.866207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.873386 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.873368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:10.967160 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.967129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.967307 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.967194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.967820 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.967798    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:10.969595 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:10.969579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls\") pod \"splitter-graph-2fdf6-7487d76957-m5hkw\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") " pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:11.172994 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:11.172893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:11.447294 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:11.447207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:11.450115 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:14:11.450092    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f96b471_c4e4_47b8_926f_77c18196acae.slice/crio-c4debd82a3cc49642ac83daa4d77930485feb947c34e2e5583636b76a4f05e31 WatchSource:0}: Error finding container c4debd82a3cc49642ac83daa4d77930485feb947c34e2e5583636b76a4f05e31: Status 404 returned error can't find the container with id c4debd82a3cc49642ac83daa4d77930485feb947c34e2e5583636b76a4f05e31
Sep 17 17:14:11.984446 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:11.984412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" event={"ID":"0f96b471-c4e4-47b8-926f-77c18196acae","Type":"ContainerStarted","Data":"4e2450d55be4d6c0e0d80d4c41a9d65db36daa03a136fb4bdfe5e38bbeac5a0f"}
Sep 17 17:14:11.984861 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:11.984463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" event={"ID":"0f96b471-c4e4-47b8-926f-77c18196acae","Type":"ContainerStarted","Data":"c4debd82a3cc49642ac83daa4d77930485feb947c34e2e5583636b76a4f05e31"}
Sep 17 17:14:11.984861 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:11.984530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:12.002680 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:12.002637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podStartSLOduration=2.002623614 podStartE2EDuration="2.002623614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:12.001179894 +0000 UTC m=+1944.698682091" watchObservedRunningTime="2026-09-17 17:14:12.002623614 +0000 UTC m=+1944.700125851"
Sep 17 17:14:14.079514 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:14.079476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:14.080008 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:14.079581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:14:17.993355 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:17.993322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:18.956053 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:18.956013    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:19.079411 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:19.079375    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:20.947265 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:20.947179    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:20.947780 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:20.947353    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" containerID="cri-o://4e2450d55be4d6c0e0d80d4c41a9d65db36daa03a136fb4bdfe5e38bbeac5a0f" gracePeriod=30
Sep 17 17:14:21.102280 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.102249    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:14:21.102530 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.102482    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container" containerID="cri-o://3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2" gracePeriod=30
Sep 17 17:14:21.102664 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.102506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kube-rbac-proxy" containerID="cri-o://f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d" gracePeriod=30
Sep 17 17:14:21.283888 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.283855    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:14:21.287491 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.287474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.289986 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.289961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2756e-predictor-serving-cert\""
Sep 17 17:14:21.290174 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.290140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2756e-kube-rbac-proxy-sar-config\""
Sep 17 17:14:21.299660 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.299637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:14:21.354300 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.354267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.354463 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.354315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.354463 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.354334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mkh\" (UniqueName: \"kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.455522 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.455489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.455693 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.455602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.455693 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.455633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mkh\" (UniqueName: \"kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.455814 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:21.455776    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2756e-predictor-serving-cert: secret "error-404-isvc-2756e-predictor-serving-cert" not found
Sep 17 17:14:21.455870 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:21.455861    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls podName:10844db6-ce4c-4da0-9119-69d934542d3b nodeName:}" failed. No retries permitted until 2026-09-17 17:14:21.955838087 +0000 UTC m=+1954.653340264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls") pod "error-404-isvc-2756e-predictor-58d849df88-k4hdz" (UID: "10844db6-ce4c-4da0-9119-69d934542d3b") : secret "error-404-isvc-2756e-predictor-serving-cert" not found
Sep 17 17:14:21.456278 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.456260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.464186 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.464168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mkh\" (UniqueName: \"kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.958840 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.958801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:21.961335 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:21.961315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") pod \"error-404-isvc-2756e-predictor-58d849df88-k4hdz\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") " pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:22.024532 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.024501    2580 generic.go:356] "Generic (PLEG): container finished" podID="15a66aa8-3b74-4249-84ab-669884df195b" containerID="f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d" exitCode=2
Sep 17 17:14:22.024704 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.024582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerDied","Data":"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"}
Sep 17 17:14:22.197634 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.197599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:22.474503 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.474476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:14:22.476859 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:14:22.476824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10844db6_ce4c_4da0_9119_69d934542d3b.slice/crio-94f5caf21052f741d01b34bd39889012355b3e81dd519ad0ecbf76d7ed97c948 WatchSource:0}: Error finding container 94f5caf21052f741d01b34bd39889012355b3e81dd519ad0ecbf76d7ed97c948: Status 404 returned error can't find the container with id 94f5caf21052f741d01b34bd39889012355b3e81dd519ad0ecbf76d7ed97c948
Sep 17 17:14:22.741012 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.740971    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 17:14:22.992139 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:22.992094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:23.030277 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:23.030235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerStarted","Data":"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"}
Sep 17 17:14:23.030465 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:23.030361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:23.030465 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:23.030394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerStarted","Data":"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"}
Sep 17 17:14:23.030465 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:23.030424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerStarted","Data":"94f5caf21052f741d01b34bd39889012355b3e81dd519ad0ecbf76d7ed97c948"}
Sep 17 17:14:23.050485 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:23.050443    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podStartSLOduration=2.050428412 podStartE2EDuration="2.050428412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:23.048571376 +0000 UTC m=+1955.746073574" watchObservedRunningTime="2026-09-17 17:14:23.050428412 +0000 UTC m=+1955.747930610"
Sep 17 17:14:24.034771 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.034740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:24.036048 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.036023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:14:24.079442 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.079412    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:24.338748 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.338721    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:14:24.378209 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.378180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2rll\" (UniqueName: \"kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll\") pod \"15a66aa8-3b74-4249-84ab-669884df195b\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") "
Sep 17 17:14:24.378379 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.378231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config\") pod \"15a66aa8-3b74-4249-84ab-669884df195b\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") "
Sep 17 17:14:24.378379 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.378261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls\") pod \"15a66aa8-3b74-4249-84ab-669884df195b\" (UID: \"15a66aa8-3b74-4249-84ab-669884df195b\") "
Sep 17 17:14:24.378563 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.378537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15a66aa8-3b74-4249-84ab-669884df195b" (UID: "15a66aa8-3b74-4249-84ab-669884df195b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:24.380452 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.380424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15a66aa8-3b74-4249-84ab-669884df195b" (UID: "15a66aa8-3b74-4249-84ab-669884df195b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:24.380573 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.380466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll" (OuterVolumeSpecName: "kube-api-access-m2rll") pod "15a66aa8-3b74-4249-84ab-669884df195b" (UID: "15a66aa8-3b74-4249-84ab-669884df195b"). InnerVolumeSpecName "kube-api-access-m2rll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:24.478745 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.478717    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rll\" (UniqueName: \"kubernetes.io/projected/15a66aa8-3b74-4249-84ab-669884df195b-kube-api-access-m2rll\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:24.478745 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.478738    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a66aa8-3b74-4249-84ab-669884df195b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:24.478745 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:24.478748    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a66aa8-3b74-4249-84ab-669884df195b-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:25.039419 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.039388    2580 generic.go:356] "Generic (PLEG): container finished" podID="15a66aa8-3b74-4249-84ab-669884df195b" containerID="3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2" exitCode=0
Sep 17 17:14:25.039934 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.039448    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"
Sep 17 17:14:25.039934 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.039479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerDied","Data":"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"}
Sep 17 17:14:25.039934 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.039546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6" event={"ID":"15a66aa8-3b74-4249-84ab-669884df195b","Type":"ContainerDied","Data":"20e746d35185496e771a73544c6588ff9790a548a85d089ffbb8d554c4a3edd2"}
Sep 17 17:14:25.039934 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.039575    2580 scope.go:117] "RemoveContainer" containerID="f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"
Sep 17 17:14:25.040244 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.040214    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:14:25.048414 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.048398    2580 scope.go:117] "RemoveContainer" containerID="3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"
Sep 17 17:14:25.056017 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.056000    2580 scope.go:117] "RemoveContainer" containerID="f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"
Sep 17 17:14:25.056266 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:25.056247    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d\": container with ID starting with f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d not found: ID does not exist" containerID="f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"
Sep 17 17:14:25.056386 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.056277    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d"} err="failed to get container status \"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d\": rpc error: code = NotFound desc = could not find container \"f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d\": container with ID starting with f37b8b5a20ca127bce87b4c288918274824526aa12e2b32a1fb991ccb2198f0d not found: ID does not exist"
Sep 17 17:14:25.056386 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.056299    2580 scope.go:117] "RemoveContainer" containerID="3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"
Sep 17 17:14:25.056565 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:25.056549    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2\": container with ID starting with 3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2 not found: ID does not exist" containerID="3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"
Sep 17 17:14:25.056604 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.056571    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2"} err="failed to get container status \"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2\": rpc error: code = NotFound desc = could not find container \"3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2\": container with ID starting with 3c5012082169154bd03242fa93b6673b3887d6cb1cc78cd521c8d024a30286c2 not found: ID does not exist"
Sep 17 17:14:25.060489 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.060468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:14:25.062257 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.062238    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2fdf6-predictor-fff5f778b-5tld6"]
Sep 17 17:14:25.363682 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:25.363585    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a66aa8-3b74-4249-84ab-669884df195b" path="/var/lib/kubelet/pods/15a66aa8-3b74-4249-84ab-669884df195b/volumes"
Sep 17 17:14:27.992470 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:27.992428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:28.956090 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:28.956053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:29.079733 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:29.079649    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:30.044793 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:30.044759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:14:30.045341 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:30.045309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:14:30.968190 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:30.968163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:14:31.031592 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.031564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") pod \"e7e38d44-028f-4511-a80d-9132da1aaa6c\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") "
Sep 17 17:14:31.031768 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.031651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle\") pod \"e7e38d44-028f-4511-a80d-9132da1aaa6c\" (UID: \"e7e38d44-028f-4511-a80d-9132da1aaa6c\") "
Sep 17 17:14:31.032012 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.031987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e7e38d44-028f-4511-a80d-9132da1aaa6c" (UID: "e7e38d44-028f-4511-a80d-9132da1aaa6c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:31.033712 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.033688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7e38d44-028f-4511-a80d-9132da1aaa6c" (UID: "e7e38d44-028f-4511-a80d-9132da1aaa6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:31.064445 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.064414    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerID="7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4" exitCode=0
Sep 17 17:14:31.064566 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.064456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" event={"ID":"e7e38d44-028f-4511-a80d-9132da1aaa6c","Type":"ContainerDied","Data":"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"}
Sep 17 17:14:31.064566 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.064465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"
Sep 17 17:14:31.064566 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.064501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j" event={"ID":"e7e38d44-028f-4511-a80d-9132da1aaa6c","Type":"ContainerDied","Data":"2407183efe3ba802c02ee8c9a9e84cec0b6579c7ad3b1224b475a2f0fd3f1e91"}
Sep 17 17:14:31.064566 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.064523    2580 scope.go:117] "RemoveContainer" containerID="7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"
Sep 17 17:14:31.073209 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.073193    2580 scope.go:117] "RemoveContainer" containerID="7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"
Sep 17 17:14:31.073511 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:14:31.073485    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4\": container with ID starting with 7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4 not found: ID does not exist" containerID="7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"
Sep 17 17:14:31.073592 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.073516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4"} err="failed to get container status \"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4\": rpc error: code = NotFound desc = could not find container \"7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4\": container with ID starting with 7981852d5145f643b77a961d2aa080222f0ab9fe530611d7b6ebc759ce442bc4 not found: ID does not exist"
Sep 17 17:14:31.084762 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.084714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:14:31.088107 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.088087    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c2858-54876bd879-lh79j"]
Sep 17 17:14:31.132274 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.132247    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e38d44-028f-4511-a80d-9132da1aaa6c-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:31.132274 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.132273    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e38d44-028f-4511-a80d-9132da1aaa6c-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:31.363775 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:31.363677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" path="/var/lib/kubelet/pods/e7e38d44-028f-4511-a80d-9132da1aaa6c/volumes"
Sep 17 17:14:32.991433 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:32.991391    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:32.991784 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:32.991495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:37.992171 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:37.992129    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:38.955455 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:38.955416    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 17:14:40.045690 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:40.045646    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:14:42.992095 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:42.992057    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:47.992673 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:47.992630    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:14:48.956586 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:48.956555    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:14:50.046371 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:50.046331    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:14:51.140210 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.140169    2580 generic.go:356] "Generic (PLEG): container finished" podID="0f96b471-c4e4-47b8-926f-77c18196acae" containerID="4e2450d55be4d6c0e0d80d4c41a9d65db36daa03a136fb4bdfe5e38bbeac5a0f" exitCode=0
Sep 17 17:14:51.140586 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.140242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" event={"ID":"0f96b471-c4e4-47b8-926f-77c18196acae","Type":"ContainerDied","Data":"4e2450d55be4d6c0e0d80d4c41a9d65db36daa03a136fb4bdfe5e38bbeac5a0f"}
Sep 17 17:14:51.587764 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.587743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:51.698731 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.698685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls\") pod \"0f96b471-c4e4-47b8-926f-77c18196acae\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") "
Sep 17 17:14:51.698933 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.698752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle\") pod \"0f96b471-c4e4-47b8-926f-77c18196acae\" (UID: \"0f96b471-c4e4-47b8-926f-77c18196acae\") "
Sep 17 17:14:51.699144 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.699110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0f96b471-c4e4-47b8-926f-77c18196acae" (UID: "0f96b471-c4e4-47b8-926f-77c18196acae"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:14:51.701020 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.700994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f96b471-c4e4-47b8-926f-77c18196acae" (UID: "0f96b471-c4e4-47b8-926f-77c18196acae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:51.799464 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.799435    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f96b471-c4e4-47b8-926f-77c18196acae-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:51.799464 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:51.799460    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f96b471-c4e4-47b8-926f-77c18196acae-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:14:52.145223 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:52.145148    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"
Sep 17 17:14:52.145223 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:52.145174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw" event={"ID":"0f96b471-c4e4-47b8-926f-77c18196acae","Type":"ContainerDied","Data":"c4debd82a3cc49642ac83daa4d77930485feb947c34e2e5583636b76a4f05e31"}
Sep 17 17:14:52.145661 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:52.145227    2580 scope.go:117] "RemoveContainer" containerID="4e2450d55be4d6c0e0d80d4c41a9d65db36daa03a136fb4bdfe5e38bbeac5a0f"
Sep 17 17:14:52.165355 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:52.165332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:52.168664 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:52.168644    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2fdf6-7487d76957-m5hkw"]
Sep 17 17:14:53.362863 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:14:53.362832    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" path="/var/lib/kubelet/pods/0f96b471-c4e4-47b8-926f-77c18196acae/volumes"
Sep 17 17:15:00.045588 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:00.045540    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 17:15:01.110543 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110806    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kube-rbac-proxy"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110818    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kube-rbac-proxy"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110848    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110855    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110861    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6"
Sep 17 17:15:01.110898 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110881    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6"
Sep 17 17:15:01.111172 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e38d44-028f-4511-a80d-9132da1aaa6c" containerName="sequence-graph-c2858"
Sep 17 17:15:01.111172 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kube-rbac-proxy"
Sep 17 17:15:01.111172 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a66aa8-3b74-4249-84ab-669884df195b" containerName="kserve-container"
Sep 17 17:15:01.111172 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.110972    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f96b471-c4e4-47b8-926f-77c18196acae" containerName="splitter-graph-2fdf6"
Sep 17 17:15:01.115162 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.115144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.117349 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.117319    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0d22-kube-rbac-proxy-sar-config\""
Sep 17 17:15:01.117349 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.117320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0d22-serving-cert\""
Sep 17 17:15:01.128450 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.128425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:15:01.276699 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.276663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.276699 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.276707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.378167 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.378092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.378167 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.378130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.378845 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.378821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.380694 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.380673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls\") pod \"switch-graph-f0d22-6c68dfbc6f-dmg57\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") " pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.425540 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.425517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:01.694363 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:01.694289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:15:01.697311 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:15:01.697285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5861dbc_73e9_4c76_bbcc_d23839e86bfa.slice/crio-7fff62dbc53751d567852b349ef244fe2f99d9750d4f94e766e6f2ef32be2527 WatchSource:0}: Error finding container 7fff62dbc53751d567852b349ef244fe2f99d9750d4f94e766e6f2ef32be2527: Status 404 returned error can't find the container with id 7fff62dbc53751d567852b349ef244fe2f99d9750d4f94e766e6f2ef32be2527
Sep 17 17:15:02.189464 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:02.189432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" event={"ID":"f5861dbc-73e9-4c76-bbcc-d23839e86bfa","Type":"ContainerStarted","Data":"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"}
Sep 17 17:15:02.189874 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:02.189480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" event={"ID":"f5861dbc-73e9-4c76-bbcc-d23839e86bfa","Type":"ContainerStarted","Data":"7fff62dbc53751d567852b349ef244fe2f99d9750d4f94e766e6f2ef32be2527"}
Sep 17 17:15:02.189874 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:02.189542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:02.211051 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:02.207023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podStartSLOduration=1.207002247 podStartE2EDuration="1.207002247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:15:02.205156088 +0000 UTC m=+1994.902658291" watchObservedRunningTime="2026-09-17 17:15:02.207002247 +0000 UTC m=+1994.904504439"
Sep 17 17:15:08.198931 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:08.198883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:15:10.046992 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:10.046958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:15:21.174117 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.174067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:15:21.177805 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.177786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.179702 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.179681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2756e-kube-rbac-proxy-sar-config\""
Sep 17 17:15:21.179785 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.179681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-2756e-serving-cert\""
Sep 17 17:15:21.188683 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.188661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:15:21.220220 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.220188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.220372 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.220254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.321236 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.321205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.321381 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.321256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.321381 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:15:21.321352    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-2756e-serving-cert: secret "splitter-graph-2756e-serving-cert" not found
Sep 17 17:15:21.321501 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:15:21.321420    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls podName:8b79fbf3-9fd0-49be-a394-e66dabcf64e4 nodeName:}" failed. No retries permitted until 2026-09-17 17:15:21.821402407 +0000 UTC m=+2014.518904585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls") pod "splitter-graph-2756e-6c865d8977-c2ssw" (UID: "8b79fbf3-9fd0-49be-a394-e66dabcf64e4") : secret "splitter-graph-2756e-serving-cert" not found
Sep 17 17:15:21.322081 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.322062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.825605 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.825556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:21.828328 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:21.828304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") pod \"splitter-graph-2756e-6c865d8977-c2ssw\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") " pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:22.087916 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:22.087821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:22.361263 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:22.361238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:15:22.363593 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:15:22.363568    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b79fbf3_9fd0_49be_a394_e66dabcf64e4.slice/crio-cb0d12a2dc4d3e4f2876b6f5a781b3e239f480027671fdf6623cc7533498f75f WatchSource:0}: Error finding container cb0d12a2dc4d3e4f2876b6f5a781b3e239f480027671fdf6623cc7533498f75f: Status 404 returned error can't find the container with id cb0d12a2dc4d3e4f2876b6f5a781b3e239f480027671fdf6623cc7533498f75f
Sep 17 17:15:23.271140 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:23.271100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" event={"ID":"8b79fbf3-9fd0-49be-a394-e66dabcf64e4","Type":"ContainerStarted","Data":"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"}
Sep 17 17:15:23.271312 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:23.271153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" event={"ID":"8b79fbf3-9fd0-49be-a394-e66dabcf64e4","Type":"ContainerStarted","Data":"cb0d12a2dc4d3e4f2876b6f5a781b3e239f480027671fdf6623cc7533498f75f"}
Sep 17 17:15:23.271312 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:23.271213    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:15:23.288037 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:23.287990    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podStartSLOduration=2.2879733079999998 podStartE2EDuration="2.287973308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:15:23.287532597 +0000 UTC m=+2015.985034796" watchObservedRunningTime="2026-09-17 17:15:23.287973308 +0000 UTC m=+2015.985475505"
Sep 17 17:15:29.281885 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:15:29.281857    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:16:49.445639 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:16:49.445610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:16:49.450062 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:16:49.450035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:16:49.450446 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:16:49.450430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:16:49.455262 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:16:49.455243    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:21:49.472717 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:21:49.472681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:21:49.477647 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:21:49.477623    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:21:49.477647 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:21:49.477635    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:21:49.482410 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:21:49.482378    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:23:35.869562 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:35.869525    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:23:35.872153 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:35.869743    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" containerID="cri-o://35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1" gracePeriod=30
Sep 17 17:23:35.995241 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:35.995190    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:23:35.995474 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:35.995410    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container" containerID="cri-o://73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd" gracePeriod=30
Sep 17 17:23:35.995547 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:35.995435    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kube-rbac-proxy" containerID="cri-o://44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec" gracePeriod=30
Sep 17 17:23:36.150463 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:36.150399    2580 generic.go:356] "Generic (PLEG): container finished" podID="10844db6-ce4c-4da0-9119-69d934542d3b" containerID="44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec" exitCode=2
Sep 17 17:23:36.150463 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:36.150451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerDied","Data":"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"}
Sep 17 17:23:38.931057 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:38.931033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:23:39.009377 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.009305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") pod \"10844db6-ce4c-4da0-9119-69d934542d3b\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") "
Sep 17 17:23:39.009377 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.009360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9mkh\" (UniqueName: \"kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh\") pod \"10844db6-ce4c-4da0-9119-69d934542d3b\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") "
Sep 17 17:23:39.009582 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.009389    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config\") pod \"10844db6-ce4c-4da0-9119-69d934542d3b\" (UID: \"10844db6-ce4c-4da0-9119-69d934542d3b\") "
Sep 17 17:23:39.009817 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.009792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10844db6-ce4c-4da0-9119-69d934542d3b" (UID: "10844db6-ce4c-4da0-9119-69d934542d3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:39.011542 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.011521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10844db6-ce4c-4da0-9119-69d934542d3b" (UID: "10844db6-ce4c-4da0-9119-69d934542d3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:23:39.011608 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.011586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh" (OuterVolumeSpecName: "kube-api-access-m9mkh") pod "10844db6-ce4c-4da0-9119-69d934542d3b" (UID: "10844db6-ce4c-4da0-9119-69d934542d3b"). InnerVolumeSpecName "kube-api-access-m9mkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:23:39.110787 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.110741    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10844db6-ce4c-4da0-9119-69d934542d3b-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:23:39.110787 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.110783    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mkh\" (UniqueName: \"kubernetes.io/projected/10844db6-ce4c-4da0-9119-69d934542d3b-kube-api-access-m9mkh\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:23:39.110787 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.110797    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10844db6-ce4c-4da0-9119-69d934542d3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:23:39.163901 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.163876    2580 generic.go:356] "Generic (PLEG): container finished" podID="10844db6-ce4c-4da0-9119-69d934542d3b" containerID="73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd" exitCode=0
Sep 17 17:23:39.164050 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.163957    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"
Sep 17 17:23:39.164050 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.163964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerDied","Data":"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"}
Sep 17 17:23:39.164050 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.164032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz" event={"ID":"10844db6-ce4c-4da0-9119-69d934542d3b","Type":"ContainerDied","Data":"94f5caf21052f741d01b34bd39889012355b3e81dd519ad0ecbf76d7ed97c948"}
Sep 17 17:23:39.164210 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.164061    2580 scope.go:117] "RemoveContainer" containerID="44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"
Sep 17 17:23:39.172582 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.172567    2580 scope.go:117] "RemoveContainer" containerID="73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"
Sep 17 17:23:39.179995 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.179980    2580 scope.go:117] "RemoveContainer" containerID="44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"
Sep 17 17:23:39.180243 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:23:39.180226    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec\": container with ID starting with 44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec not found: ID does not exist" containerID="44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"
Sep 17 17:23:39.180294 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.180252    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec"} err="failed to get container status \"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec\": rpc error: code = NotFound desc = could not find container \"44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec\": container with ID starting with 44a46b23d7c107ba9b8c3ea6a8821b4c5faa277b8cd196f0e4f709e8253b47ec not found: ID does not exist"
Sep 17 17:23:39.180294 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.180268    2580 scope.go:117] "RemoveContainer" containerID="73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"
Sep 17 17:23:39.180491 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:23:39.180474    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd\": container with ID starting with 73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd not found: ID does not exist" containerID="73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"
Sep 17 17:23:39.180534 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.180494    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd"} err="failed to get container status \"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd\": rpc error: code = NotFound desc = could not find container \"73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd\": container with ID starting with 73a8c150dfed17a217c0defb7902424ce18b8e67cd28350d64209f9bc1eafddd not found: ID does not exist"
Sep 17 17:23:39.183666 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.183644    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:23:39.187119 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.187099    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2756e-predictor-58d849df88-k4hdz"]
Sep 17 17:23:39.279101 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.279021    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:23:39.363448 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:39.363418    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" path="/var/lib/kubelet/pods/10844db6-ce4c-4da0-9119-69d934542d3b/volumes"
Sep 17 17:23:44.279409 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:44.279374    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:23:49.279755 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:49.279717    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:23:49.280153 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:49.279817    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:23:54.279507 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:54.279470    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:23:59.279289 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:23:59.279250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:24:04.279628 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:04.279591    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:24:06.016501 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.016474    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:24:06.018532 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.018511    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle\") pod \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") "
Sep 17 17:24:06.018641 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.018589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") pod \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\" (UID: \"8b79fbf3-9fd0-49be-a394-e66dabcf64e4\") "
Sep 17 17:24:06.018861 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.018838    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8b79fbf3-9fd0-49be-a394-e66dabcf64e4" (UID: "8b79fbf3-9fd0-49be-a394-e66dabcf64e4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:24:06.020837 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.020816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b79fbf3-9fd0-49be-a394-e66dabcf64e4" (UID: "8b79fbf3-9fd0-49be-a394-e66dabcf64e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:24:06.119884 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.119856    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:24:06.119884 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.119879    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b79fbf3-9fd0-49be-a394-e66dabcf64e4-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:24:06.274781 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.274700    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerID="35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1" exitCode=0
Sep 17 17:24:06.274781 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.274749    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"
Sep 17 17:24:06.274781 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.274772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" event={"ID":"8b79fbf3-9fd0-49be-a394-e66dabcf64e4","Type":"ContainerDied","Data":"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"}
Sep 17 17:24:06.275065 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.274813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw" event={"ID":"8b79fbf3-9fd0-49be-a394-e66dabcf64e4","Type":"ContainerDied","Data":"cb0d12a2dc4d3e4f2876b6f5a781b3e239f480027671fdf6623cc7533498f75f"}
Sep 17 17:24:06.275065 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.274834    2580 scope.go:117] "RemoveContainer" containerID="35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"
Sep 17 17:24:06.283435 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.283420    2580 scope.go:117] "RemoveContainer" containerID="35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"
Sep 17 17:24:06.283671 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:24:06.283654    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1\": container with ID starting with 35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1 not found: ID does not exist" containerID="35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"
Sep 17 17:24:06.283725 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.283678    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1"} err="failed to get container status \"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1\": rpc error: code = NotFound desc = could not find container \"35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1\": container with ID starting with 35178313165f1bdc17c3c4376595dcaed35334a4ded23c9eb5d13dfc3f64efd1 not found: ID does not exist"
Sep 17 17:24:06.293310 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.293290    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:24:06.296766 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:06.296746    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-2756e-6c865d8977-c2ssw"]
Sep 17 17:24:07.362901 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:24:07.362871    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" path="/var/lib/kubelet/pods/8b79fbf3-9fd0-49be-a394-e66dabcf64e4/volumes"
Sep 17 17:26:49.496027 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:26:49.495995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:26:49.500892 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:26:49.500874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:26:49.502179 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:26:49.502163    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:26:49.508141 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:26:49.508122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:31:20.327390 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.327302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:31:20.329773 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.327514    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" containerID="cri-o://8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854" gracePeriod=30
Sep 17 17:31:20.726789 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.726696    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:31:20.727111 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.727077    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container" containerID="cri-o://a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000" gracePeriod=30
Sep 17 17:31:20.727387 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.727354    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kube-rbac-proxy" containerID="cri-o://a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c" gracePeriod=30
Sep 17 17:31:20.897920 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.897881    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1621874-4f05-4d26-9144-22a9ad3db714" containerID="a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c" exitCode=2
Sep 17 17:31:20.898078 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:20.897936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerDied","Data":"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"}
Sep 17 17:31:23.197364 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.197329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:23.464626 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.464602    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:31:23.611378 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.611288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config\") pod \"e1621874-4f05-4d26-9144-22a9ad3db714\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") "
Sep 17 17:31:23.611378 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.611336    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") pod \"e1621874-4f05-4d26-9144-22a9ad3db714\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") "
Sep 17 17:31:23.611378 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.611354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phvws\" (UniqueName: \"kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws\") pod \"e1621874-4f05-4d26-9144-22a9ad3db714\" (UID: \"e1621874-4f05-4d26-9144-22a9ad3db714\") "
Sep 17 17:31:23.611738 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.611713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1621874-4f05-4d26-9144-22a9ad3db714" (UID: "e1621874-4f05-4d26-9144-22a9ad3db714"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:31:23.613606 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.613582    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1621874-4f05-4d26-9144-22a9ad3db714" (UID: "e1621874-4f05-4d26-9144-22a9ad3db714"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:31:23.613718 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.613623    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws" (OuterVolumeSpecName: "kube-api-access-phvws") pod "e1621874-4f05-4d26-9144-22a9ad3db714" (UID: "e1621874-4f05-4d26-9144-22a9ad3db714"). InnerVolumeSpecName "kube-api-access-phvws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:31:23.712132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.712104    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1621874-4f05-4d26-9144-22a9ad3db714-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:31:23.712132 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.712130    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1621874-4f05-4d26-9144-22a9ad3db714-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:31:23.712296 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.712141    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvws\" (UniqueName: \"kubernetes.io/projected/e1621874-4f05-4d26-9144-22a9ad3db714-kube-api-access-phvws\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:31:23.914583 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.914482    2580 generic.go:356] "Generic (PLEG): container finished" podID="e1621874-4f05-4d26-9144-22a9ad3db714" containerID="a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000" exitCode=0
Sep 17 17:31:23.914583 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.914538    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"
Sep 17 17:31:23.914808 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.914577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerDied","Data":"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"}
Sep 17 17:31:23.914808 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.914652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb" event={"ID":"e1621874-4f05-4d26-9144-22a9ad3db714","Type":"ContainerDied","Data":"727d8ae96a5c250ba266e418ba58bd684038eb2a8d4c602c54f60294420f840e"}
Sep 17 17:31:23.914808 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.914716    2580 scope.go:117] "RemoveContainer" containerID="a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"
Sep 17 17:31:23.923384 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.923365    2580 scope.go:117] "RemoveContainer" containerID="a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"
Sep 17 17:31:23.930380 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.930362    2580 scope.go:117] "RemoveContainer" containerID="a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"
Sep 17 17:31:23.930637 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:31:23.930619    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c\": container with ID starting with a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c not found: ID does not exist" containerID="a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"
Sep 17 17:31:23.930707 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.930646    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c"} err="failed to get container status \"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c\": rpc error: code = NotFound desc = could not find container \"a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c\": container with ID starting with a977413e838133041932d1cfd2af777566c89a92c459a8e52be1e1987de4fa2c not found: ID does not exist"
Sep 17 17:31:23.930707 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.930661    2580 scope.go:117] "RemoveContainer" containerID="a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"
Sep 17 17:31:23.930902 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:31:23.930884    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000\": container with ID starting with a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000 not found: ID does not exist" containerID="a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"
Sep 17 17:31:23.930902 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.930925    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000"} err="failed to get container status \"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000\": rpc error: code = NotFound desc = could not find container \"a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000\": container with ID starting with a5678ac4fbdc2c78f4c998475f387fff9823a1fdf7d301c48a2e8f22ce1a9000 not found: ID does not exist"
Sep 17 17:31:23.953716 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.953691    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:31:23.959782 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:23.959762    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f0d22-predictor-7667c7cb86-cqmgb"]
Sep 17 17:31:25.363243 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:25.363208    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" path="/var/lib/kubelet/pods/e1621874-4f05-4d26-9144-22a9ad3db714/volumes"
Sep 17 17:31:28.197516 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:28.197475    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:33.197175 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:33.197130    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:33.197618 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:33.197250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:31:37.590557 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:37.590526    2580 ???:1] "http: TLS handshake error from 10.0.143.228:35202: EOF"
Sep 17 17:31:37.594010 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:37.593985    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:38.197090 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:38.197058    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:38.395984 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:38.395953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:39.178228 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:39.178200    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:39.942506 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:39.942471    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:40.711495 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:40.711451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:41.464958 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:41.464930    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:42.220276 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:42.220248    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:43.005925 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:43.005874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:43.197238 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:43.197202    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:43.776301 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:43.776268    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:44.582011 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:44.581986    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:45.363921 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:45.363877    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:46.172859 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:46.172827    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:46.957183 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:46.957153    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:47.733022 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:47.732993    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:48.197343 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:48.197304    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:31:48.531528 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:48.531433    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-f0d22-6c68dfbc6f-dmg57_f5861dbc-73e9-4c76-bbcc-d23839e86bfa/switch-graph-f0d22/0.log"
Sep 17 17:31:49.518883 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:49.518853    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:31:49.523780 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:49.523756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:31:49.527848 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:49.527829    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:31:49.532599 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:49.532582    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:31:50.473367 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.473345    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:31:50.502716 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.502693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle\") pod \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") "
Sep 17 17:31:50.502857 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.502742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls\") pod \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\" (UID: \"f5861dbc-73e9-4c76-bbcc-d23839e86bfa\") "
Sep 17 17:31:50.503148 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.503121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f5861dbc-73e9-4c76-bbcc-d23839e86bfa" (UID: "f5861dbc-73e9-4c76-bbcc-d23839e86bfa"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:31:50.505012 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.504992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5861dbc-73e9-4c76-bbcc-d23839e86bfa" (UID: "f5861dbc-73e9-4c76-bbcc-d23839e86bfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:31:50.604213 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.604140    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-openshift-service-ca-bundle\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:31:50.604213 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:50.604167    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5861dbc-73e9-4c76-bbcc-d23839e86bfa-proxy-tls\") on node \"ip-10-0-143-112.ec2.internal\" DevicePath \"\""
Sep 17 17:31:51.012475 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.012400    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerID="8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854" exitCode=0
Sep 17 17:31:51.012475 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.012438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"
Sep 17 17:31:51.012475 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.012463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" event={"ID":"f5861dbc-73e9-4c76-bbcc-d23839e86bfa","Type":"ContainerDied","Data":"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"}
Sep 17 17:31:51.012821 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.012504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57" event={"ID":"f5861dbc-73e9-4c76-bbcc-d23839e86bfa","Type":"ContainerDied","Data":"7fff62dbc53751d567852b349ef244fe2f99d9750d4f94e766e6f2ef32be2527"}
Sep 17 17:31:51.012821 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.012525    2580 scope.go:117] "RemoveContainer" containerID="8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"
Sep 17 17:31:51.023155 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.023137    2580 scope.go:117] "RemoveContainer" containerID="8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"
Sep 17 17:31:51.023399 ip-10-0-143-112 kubenswrapper[2580]: E0917 17:31:51.023380    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854\": container with ID starting with 8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854 not found: ID does not exist" containerID="8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"
Sep 17 17:31:51.023453 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.023409    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854"} err="failed to get container status \"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854\": rpc error: code = NotFound desc = could not find container \"8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854\": container with ID starting with 8c5dd4339cd0f24e43ef77171e899da8509cbcca550323691ba058fa761a1854 not found: ID does not exist"
Sep 17 17:31:51.031797 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.031774    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:31:51.035418 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.035396    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0d22-6c68dfbc6f-dmg57"]
Sep 17 17:31:51.362627 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:51.362594    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" path="/var/lib/kubelet/pods/f5861dbc-73e9-4c76-bbcc-d23839e86bfa/volumes"
Sep 17 17:31:56.307069 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:56.307042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzlws_2b13d7a0-ed59-45f9-8ec6-6c9f9c69ad43/global-pull-secret-syncer/0.log"
Sep 17 17:31:56.408015 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:56.407980    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xqzcq_c2097712-860e-4e0a-85ef-f699c9d02b3f/konnectivity-agent/0.log"
Sep 17 17:31:56.454309 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:31:56.454284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-112.ec2.internal_607b69b0e690d92464a627a4aea229b6/haproxy/0.log"
Sep 17 17:32:00.648155 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:00.648125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-lktl5_c53fff2f-d07b-4ecf-a1b8-3d55e635f7ec/monitoring-plugin/0.log"
Sep 17 17:32:00.748515 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:00.748491    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpnz9_6f103dfe-af4d-4c2c-8d68-64f60caa6ab7/node-exporter/0.log"
Sep 17 17:32:00.772121 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:00.772095    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpnz9_6f103dfe-af4d-4c2c-8d68-64f60caa6ab7/kube-rbac-proxy/0.log"
Sep 17 17:32:00.801408 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:00.801384    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fpnz9_6f103dfe-af4d-4c2c-8d68-64f60caa6ab7/init-textfile/0.log"
Sep 17 17:32:03.445993 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:03.445961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/1.log"
Sep 17 17:32:03.450425 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:03.450404    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-f9cwr_0ff77b6f-6fb4-4b00-ac5a-dc4148714ac5/console-operator/2.log"
Sep 17 17:32:03.871019 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:03.870991    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-858899875d-56fmv_c91285f1-8c3d-407b-a6db-1f3d1e902db7/console/0.log"
Sep 17 17:32:04.446917 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.446874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-lvfdc_5eef0184-be01-43ad-a56d-3f5819f04916/volume-data-source-validator/0.log"
Sep 17 17:32:04.790206 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790177    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"]
Sep 17 17:32:04.790449 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790436    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790451    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790463    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790469    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790479    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790484    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790493    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container"
Sep 17 17:32:04.790504 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790498    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790508    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790514    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790523    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790528    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790571    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790578    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kserve-container"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790583    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1621874-4f05-4d26-9144-22a9ad3db714" containerName="kserve-container"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790588    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10844db6-ce4c-4da0-9119-69d934542d3b" containerName="kube-rbac-proxy"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790593    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5861dbc-73e9-4c76-bbcc-d23839e86bfa" containerName="switch-graph-f0d22"
Sep 17 17:32:04.790728 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.790597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b79fbf3-9fd0-49be-a394-e66dabcf64e4" containerName="splitter-graph-2756e"
Sep 17 17:32:04.794684 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.794665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:04.796640 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.796619    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6kqg\"/\"openshift-service-ca.crt\""
Sep 17 17:32:04.796742 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.796617    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6kqg\"/\"kube-root-ca.crt\""
Sep 17 17:32:04.796742 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.796629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6kqg\"/\"default-dockercfg-gk9tj\""
Sep 17 17:32:04.803469 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.803446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"]
Sep 17 17:32:04.907133 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.907101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-sys\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:04.907133 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.907136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdxs\" (UniqueName: \"kubernetes.io/projected/921f39d3-67da-427a-b649-9a8f25574c2a-kube-api-access-5rdxs\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:04.907312 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.907196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-lib-modules\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:04.907312 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.907264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-podres\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:04.907312 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:04.907300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-proc\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008571 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-sys\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008701 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdxs\" (UniqueName: \"kubernetes.io/projected/921f39d3-67da-427a-b649-9a8f25574c2a-kube-api-access-5rdxs\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008701 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-lib-modules\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008701 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-podres\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008701 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-sys\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008701 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-proc\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008939 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-proc\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008939 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-lib-modules\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.008939 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.008816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/921f39d3-67da-427a-b649-9a8f25574c2a-podres\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.017153 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.017136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdxs\" (UniqueName: \"kubernetes.io/projected/921f39d3-67da-427a-b649-9a8f25574c2a-kube-api-access-5rdxs\") pod \"perf-node-gather-daemonset-rn4kq\" (UID: \"921f39d3-67da-427a-b649-9a8f25574c2a\") " pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.103757 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.103699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:05.376041 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.375820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"]
Sep 17 17:32:05.380351 ip-10-0-143-112 kubenswrapper[2580]: W0917 17:32:05.380323    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod921f39d3_67da_427a_b649_9a8f25574c2a.slice/crio-3019351dd11453f99a11b794f5757ffce66487cabd0122ec22a0748547a47aeb WatchSource:0}: Error finding container 3019351dd11453f99a11b794f5757ffce66487cabd0122ec22a0748547a47aeb: Status 404 returned error can't find the container with id 3019351dd11453f99a11b794f5757ffce66487cabd0122ec22a0748547a47aeb
Sep 17 17:32:05.423551 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.423534    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q49bh_095872e5-4dd2-4526-abec-91cea5f04f2d/dns/0.log"
Sep 17 17:32:05.445107 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.445089    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q49bh_095872e5-4dd2-4526-abec-91cea5f04f2d/kube-rbac-proxy/0.log"
Sep 17 17:32:05.575003 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:05.574981    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z24x4_d6e802b9-240f-4828-b1af-e66cc459d335/dns-node-resolver/0.log"
Sep 17 17:32:06.058552 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.058527    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cb59b98fb-4jxhj_549fa0e2-b96d-4c2d-bb79-713e482c6c06/registry/0.log"
Sep 17 17:32:06.068534 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.068508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq" event={"ID":"921f39d3-67da-427a-b649-9a8f25574c2a","Type":"ContainerStarted","Data":"69111eeb82dc8c4ddcea5daa1cda0e9389baaba69169ce2740ffa67b71a2a7df"}
Sep 17 17:32:06.068683 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.068558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq" event={"ID":"921f39d3-67da-427a-b649-9a8f25574c2a","Type":"ContainerStarted","Data":"3019351dd11453f99a11b794f5757ffce66487cabd0122ec22a0748547a47aeb"}
Sep 17 17:32:06.068683 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.068602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:06.077473 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.077455    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4cz9d_db51668a-9f53-4ee9-b8c3-1d0a9aaf826f/node-ca/0.log"
Sep 17 17:32:06.084101 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:06.084066    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq" podStartSLOduration=2.084055164 podStartE2EDuration="2.084055164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:32:06.083020447 +0000 UTC m=+3018.780522646" watchObservedRunningTime="2026-09-17 17:32:06.084055164 +0000 UTC m=+3018.781557394"
Sep 17 17:32:07.269397 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.269371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5hds_32b1767d-f4fe-4c8c-8cc6-c23f21dc5fe7/serve-healthcheck-canary/0.log"
Sep 17 17:32:07.647688 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.647662    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sd2q8_6073644f-97cc-4086-a8da-3b606a68d09b/insights-operator/0.log"
Sep 17 17:32:07.648550 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.648535    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sd2q8_6073644f-97cc-4086-a8da-3b606a68d09b/insights-operator/1.log"
Sep 17 17:32:07.762719 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.762700    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dmv8b_99e8e5c3-cab1-4728-ab09-30f5987ca1bc/kube-rbac-proxy/0.log"
Sep 17 17:32:07.786020 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.785999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dmv8b_99e8e5c3-cab1-4728-ab09-30f5987ca1bc/exporter/0.log"
Sep 17 17:32:07.809752 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:07.809728    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dmv8b_99e8e5c3-cab1-4728-ab09-30f5987ca1bc/extractor/0.log"
Sep 17 17:32:12.081647 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:12.081620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6kqg/perf-node-gather-daemonset-rn4kq"
Sep 17 17:32:15.028147 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:15.028058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-mskw4_b50784ed-9d9d-440c-8075-b24d6b16a4fb/kube-storage-version-migrator-operator/1.log"
Sep 17 17:32:15.028854 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:15.028836    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-mskw4_b50784ed-9d9d-440c-8075-b24d6b16a4fb/kube-storage-version-migrator-operator/0.log"
Sep 17 17:32:16.367434 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.367408    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/kube-multus-additional-cni-plugins/0.log"
Sep 17 17:32:16.387832 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.387811    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/egress-router-binary-copy/0.log"
Sep 17 17:32:16.411058 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.411038    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/cni-plugins/0.log"
Sep 17 17:32:16.432353 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.432333    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/bond-cni-plugin/0.log"
Sep 17 17:32:16.451868 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.451845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/routeoverride-cni/0.log"
Sep 17 17:32:16.481793 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.481775    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/whereabouts-cni-bincopy/0.log"
Sep 17 17:32:16.503178 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.503164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx7rd_fc823426-4ef1-4d6e-86be-aff70a072c07/whereabouts-cni/0.log"
Sep 17 17:32:16.574086 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.574064    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c7z4p_40ca1de2-f941-4a8f-80b7-b413cf381930/kube-multus/0.log"
Sep 17 17:32:16.675416 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.675325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k8mwq_9336b6a4-be76-4751-92fa-9372275f0ddd/network-metrics-daemon/0.log"
Sep 17 17:32:16.698856 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:16.698834    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k8mwq_9336b6a4-be76-4751-92fa-9372275f0ddd/kube-rbac-proxy/0.log"
Sep 17 17:32:17.943052 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:17.943027    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-controller/0.log"
Sep 17 17:32:17.966768 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:17.966743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/0.log"
Sep 17 17:32:17.966943 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:17.966795    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovn-acl-logging/1.log"
Sep 17 17:32:17.985454 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:17.985423    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/kube-rbac-proxy-node/0.log"
Sep 17 17:32:18.005773 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:18.005744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 17:32:18.027497 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:18.027467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/northd/0.log"
Sep 17 17:32:18.047689 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:18.047664    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/nbdb/0.log"
Sep 17 17:32:18.069384 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:18.069364    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/sbdb/0.log"
Sep 17 17:32:18.111323 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:18.111300    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4plf_f8617572-e20f-4969-ac9d-b6e375107c96/ovnkube-controller/0.log"
Sep 17 17:32:19.261192 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:19.261154    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jznpc_116f607f-61d0-4a92-9045-d7e5b6cba679/network-check-target-container/0.log"
Sep 17 17:32:20.199185 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:20.199155    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wrxq6_17ec5e11-8ebd-4267-bc43-f147545d2b58/iptables-alerter/0.log"
Sep 17 17:32:20.943213 ip-10-0-143-112 kubenswrapper[2580]: I0917 17:32:20.943184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wjl99_81fac588-8726-472a-8b17-249a1384ddbc/tuned/0.log"