Sep 16 18:43:27.803057 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:43:27.803188 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:43:27.803285 ip-10-0-137-226 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:43:27.803582 ip-10-0-137-226 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:43:38.035439 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:43:38.035456 ip-10-0-137-226 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c06efbb5f7ed45a6a768938849f86cf4 --
Sep 16 18:45:11.153081 ip-10-0-137-226 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:11.519233 ip-10-0-137-226 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:11.519233 ip-10-0-137-226 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:11.519233 ip-10-0-137-226 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:11.519233 ip-10-0-137-226 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:11.519233 ip-10-0-137-226 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:11.520494 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.520432    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:11.523925 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523910    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.523925 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523925    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523929    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523932    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523935    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523940    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523945    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523948    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523951    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523953    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523956    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523959    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523962    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523964    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523967    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523970    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523972    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523975    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523978    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523980    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523983    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.523991 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523985    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523988    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523991    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523994    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523997    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.523999    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524002    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524005    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524015    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524019    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524022    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524025    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524027    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524030    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524035    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524038    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524040    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524043    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524046    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.524461 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524048    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524051    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524054    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524057    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524059    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524062    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524065    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524067    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524070    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524073    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524075    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524078    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524081    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524083    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524087    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524090    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524092    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524095    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524099    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.525046 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524103    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524107    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524110    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524112    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524116    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524120    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524122    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524125    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524128    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524130    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524133    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524136    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524138    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524141    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524144    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524146    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524149    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524152    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524154    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.525542 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524157    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524160    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524163    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524165    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524168    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524170    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524173    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524175    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524179    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524182    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524184    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524187    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524191    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524194    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524196    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524199    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524202    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524204    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524207    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524210    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.526013 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524859    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524865    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524868    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524871    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524873    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524876    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524879    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524882    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524885    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524887    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524890    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524893    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524895    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524898    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524901    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524903    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524906    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524908    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524911    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524914    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.526487 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524917    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524920    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524923    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524925    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524928    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524931    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524933    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524936    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524939    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524941    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524944    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524946    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524949    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524951    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524954    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524956    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524959    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524962    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524964    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524967    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.526996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524969    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524972    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524974    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524977    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524979    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524982    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524984    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524987    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524990    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524993    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524995    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.524998    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525003    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525006    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525008    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525011    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525013    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525016    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525019    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525021    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.527491 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525025    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525028    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525030    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525033    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525037    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525041    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525045    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525048    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525052    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525055    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525058    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525061    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525063    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525066    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525069    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525071    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525074    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525076    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525079    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.527992 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525081    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525084    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525087    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525090    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525093    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525096    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525099    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525101    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525104    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525107    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525111    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525113    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525116    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525120    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525124    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525126    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525129    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525132    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525135    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525138    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.528460 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525528    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525533    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525536    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525539    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525542    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525545    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525548    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525551    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525554    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525556    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525560    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525564    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525567    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525570    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525573    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525576    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525579    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525582    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525585    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.528976 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525588    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525591    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525593    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525596    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525599    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525601    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525604    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525606    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525609    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525611    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525614    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525617    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525619    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525622    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525624    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525627    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525630    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525632    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525636    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525639    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.529432 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525642    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525644    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525647    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525650    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525652    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525655    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525657    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525663    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525666    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525668    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525672    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525675    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525678    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525680    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525683    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525685    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525688    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525690    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525693    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525695    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.529928 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525697    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525700    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525702    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525705    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525707    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525710    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525713    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525715    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525718    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525721    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525723    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525726    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525728    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525731    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525734    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525736    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525739    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525742    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525744    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525747    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.530444 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525755    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525758    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525761    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525763    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525766    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525769    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525771    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525774    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525778    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525782    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525785    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525787    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525790    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525792    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525795    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525798    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525800    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525803    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525805    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.525808    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.530934 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525882    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525920    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525933    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525941    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525945    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525948    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525954    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525958    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525961    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525964    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525968    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525971    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525974    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525983    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525988    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525991    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525994    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.525997    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526004    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526007    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526010    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526013    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526016    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526020    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:11.531445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526023    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526026    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526029    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526032    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526035    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526038    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526041    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526044    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526048    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526050    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526053    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526056    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526059    2563 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526062    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526071    2563 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526074    2563 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526077    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526080    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526083    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526086    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526089    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526092    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526102    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526105    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526108    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:11.532068 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526111    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526114    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526117    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526120    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526122    2563 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526126    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526129    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526132    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526135    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526138    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526141    2563 flags.go:64] FLAG: --help="false"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526144    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526147    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526150    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526152    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526156    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526159    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526162    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526165    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526167    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526170    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526175    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526178    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526181    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:11.532740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526184    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526186    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526189    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526192    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526195    2563 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526199    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526209    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526212    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526217    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526220    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526223    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526226    2563 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526229    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526232    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526235    2563 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526238    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526242    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526244    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526248    2563 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526251    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526254    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526257    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526259    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526262    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526265    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:11.533403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526268    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526275    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526277    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526280    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526284    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526287    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526292    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526295    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526298    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526301    2563 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526304    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526307    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb421684165ca34e"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526311    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526314    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526323    2563 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526326    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526329    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526332    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526335    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526338    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526341    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526344    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526347    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526350    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526353    2563 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:11.534137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526356    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526359    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526362    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526365    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526367    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526370    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526373    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526376    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526379    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526381    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526384    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526387    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526391    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526394    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526397    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526402    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526404    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526407    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526413    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526418    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526421    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526424    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526432    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526435    2563 flags.go:64] FLAG: --v="2"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526439    2563 flags.go:64] FLAG: --version="false"
Sep 16 18:45:11.534800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526443    2563 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526446    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.526449    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526596    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526601    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526604    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526608    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526611    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526614    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526617    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526620    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526623    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526626    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526629    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526631    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526634    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526637    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526639    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526642    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526646    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.535521 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526650    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526653    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526657    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526659    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526662    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526665    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526669    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526671    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526674    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526676    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526679    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526682    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526685    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526688    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526690    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526693    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526696    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526698    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526701    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526703    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.536082 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526706    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526709    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526711    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526714    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526716    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526719    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526722    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526724    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526727    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526730    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526732    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526736    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526739    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526742    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526744    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526747    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526749    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526752    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526756    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526758    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.536584 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526761    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526763    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526766    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526769    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526772    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526775    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526778    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526780    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526783    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526785    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526788    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526790    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526793    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526795    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526798    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526800    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526803    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526805    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526808    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526810    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.537097 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526825    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526827    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526831    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526836    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526838    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526841    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526844    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526846    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526849    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526851    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526855    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526859    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526863    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526866    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526869    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526871    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526874    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526877    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526880    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.537613 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526882    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526885    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.526887    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.527509    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.533533    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.533547    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.534259    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.534267    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534329    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534336    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534340    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534343    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534346    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534350    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.538096 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534353    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534356    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534359    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534363    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534367    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534370    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534373    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534376    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534379    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534381    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534384    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534386    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534389    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534392    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534394    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534405    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534408    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534411    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534413    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534416    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.538441 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534418    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534421    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534424    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534427    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534430    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534433    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534436    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534440    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534443    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534447    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534450    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534453    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534456    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534458    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534461    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534464    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534467    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534470    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534473    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.539006 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534475    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534478    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534481    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534484    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534486    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534489    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534491    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534494    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534496    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534506    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534509    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534512    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534514    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534517    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534519    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534522    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534526    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534528    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534531    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534534    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.539484 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534536    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534539    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534541    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534544    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534546    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534549    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534552    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534554    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534557    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534559    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534562    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534564    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534567    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534569    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534572    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534574    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534577    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534580    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534582    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534585    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.539996 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534587    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534593    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534601    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534604    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534607    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534609    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534612    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534614    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534617    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534620    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534623    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534625    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534628    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534630    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.534635    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:11.540465 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534777    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534782    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534785    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534788    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534791    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534794    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534797    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534800    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534803    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534806    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534809    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534812    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534831    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534834    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534836    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534839    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534842    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534844    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534847    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534850    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:11.540855 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534859    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534862    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534864    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534867    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534870    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534874    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534878    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534881    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534884    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534887    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534889    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534892    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534894    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534897    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534899    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534902    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534904    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534907    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534911    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:11.541351 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534914    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534917    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534920    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534923    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534926    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534929    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534931    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534934    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534937    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534940    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534942    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534945    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534947    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534950    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534958    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534960    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534963    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534965    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534968    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534971    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:11.541828 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534973    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534976    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534978    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534981    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534984    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534986    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534989    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534992    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534994    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534997    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.534999    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535002    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535005    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535007    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535010    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535013    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535015    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535017    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535020    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:11.542335 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535023    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535025    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535028    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535030    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535033    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535035    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535038    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535040    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535048    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535051    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535054    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535056    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535059    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535062    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535066    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535069    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535072    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535074    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535077    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535079    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:11.542788 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:11.535082    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.535092    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.535209    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.538592    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.539339    2563 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.539443    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.539485    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:11.543322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.542070    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:11.550541 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.550522    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:11.558794 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.558778    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:11.562795 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.562779    2563 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:11.563661 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.563646    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:11.567909 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.567883    2563 fs.go:135] Filesystem UUIDs: map[0e986d9b-e122-4aaa-a18b-161cd424d2a5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ad2a3cdc-96a0-4431-85ae-ca7f9ad8e7f5:/dev/nvme0n1p3]
Sep 16 18:45:11.567994 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.567907    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:11.573066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.572957    2563 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:11.571833535 +0000 UTC m=+0.310409023 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099084 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21105b360eb3c5845b9049af2c1ae6 SystemUUID:ec21105b-360e-b3c5-845b-9049af2c1ae6 BootID:c06efbb5-f7ed-45a6-a768-938849f86cf4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:51:fc:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:51:fc:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:88:e5:b0:f6:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:11.573066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.573055    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:11.573220 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.573146    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:11.573988 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.573971    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:11.576942 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.576920    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:11.577100 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.576944    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:11.578040 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.578030    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:11.578078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.578043    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:11.578078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.578050    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:11.578653 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.578643    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:11.579510 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.579500    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:11.583490 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583479    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:11.583610 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583601    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:11.583639 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583631    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:11.583666 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583647    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:11.583666 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583657    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:11.583726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583666    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:11.583726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.583686    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:11.584132 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.584119    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:11.584206 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.584138    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:11.587336 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.587315    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:11.590602 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.590587    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:11.590664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.590617    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:11.593324 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593309    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593327    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593334    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593340    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593348    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593353    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593367    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593373    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593380    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593385    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:11.593391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593393    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:11.593647 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593398    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:11.593942 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593933    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:11.593942 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.593942    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:11.594650 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.594625    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:11.594650 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.594624    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:11.597394 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.597380    2563 server.go:1268] "Started kubelet"
Sep 16 18:45:11.597497 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.597459    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:11.597556 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.597485    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:11.597606 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.597567    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:11.598239 ip-10-0-137-226 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:11.599322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.598520    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:11.599322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.599040    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:11.599487 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.599464    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:11.604140 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.604121    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:11.605766 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.605666    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:11.606155 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.606140    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:11.606732 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.606341    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:11.607013 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.606981    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:11.607013 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607012    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:11.607159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607021    2563 factory.go:55] Registering systemd factory
Sep 16 18:45:11.607159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607027    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:11.607159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607035    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:11.607159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607119    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:11.607159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607127    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:11.607398 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607311    2563 factory.go:153] Registering CRI-O factory
Sep 16 18:45:11.607398 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607325    2563 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:11.607398 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607377    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:11.607398 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607394    2563 factory.go:103] Registering Raw factory
Sep 16 18:45:11.607525 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607406    2563 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:11.607569 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.606730    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-226.ec2.internal.18d5e173245f4ef7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-226.ec2.internal,UID:ip-10-0-137-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-226.ec2.internal,},FirstTimestamp:2026-09-16 18:45:11.597354743 +0000 UTC m=+0.335930232,LastTimestamp:2026-09-16 18:45:11.597354743 +0000 UTC m=+0.335930232,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-226.ec2.internal,}"
Sep 16 18:45:11.607830 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.607802    2563 manager.go:349] Starting recovery of all containers
Sep 16 18:45:11.610804 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.610778    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:11.610916 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.610804    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:11.620114 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.619962    2563 manager.go:354] Recovery completed
Sep 16 18:45:11.624956 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.624941    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.626675 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.626658    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.626743 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.626686    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.626743 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.626696    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.627223 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.627203    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:11.627223 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.627221    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:11.627323 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.627235    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:11.627323 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.627213    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.628000 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.627982    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.628092 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.628014    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.628092 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.628028    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.628802 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.628784    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f2dth"
Sep 16 18:45:11.629233 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.629141    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-226.ec2.internal.18d5e173261ead19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-226.ec2.internal,UID:ip-10-0-137-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-226.ec2.internal,},FirstTimestamp:2026-09-16 18:45:11.626673433 +0000 UTC m=+0.365248923,LastTimestamp:2026-09-16 18:45:11.626673433 +0000 UTC m=+0.365248923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-226.ec2.internal,}"
Sep 16 18:45:11.629408 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.629396    2563 policy_none.go:49] "None policy: Start"
Sep 16 18:45:11.629408 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.629412    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:11.629514 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.629422    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:11.631397 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.631385    2563 policy_none.go:47] "Start"
Sep 16 18:45:11.637606 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.637587    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f2dth"
Sep 16 18:45:11.640196 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.640119    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-226.ec2.internal.18d5e173261ef181  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-226.ec2.internal,UID:ip-10-0-137-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-226.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-226.ec2.internal,},FirstTimestamp:2026-09-16 18:45:11.626690945 +0000 UTC m=+0.365266435,LastTimestamp:2026-09-16 18:45:11.626690945 +0000 UTC m=+0.365266435,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-226.ec2.internal,}"
Sep 16 18:45:11.653431 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653418    2563 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:11.653501 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.653453    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:11.653501 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653462    2563 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:11.653731 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653717    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:11.653831 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653733    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:11.654090 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653896    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:11.654090 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653968    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:11.654090 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.653976    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:11.654581 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.654558    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:11.654663 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.654614    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:11.716115 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.716081    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:11.717264 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.717249    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:11.717354 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.717302    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:11.717354 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.717326    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:11.717458 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.717396    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:11.719645 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.719624    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:11.754225 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.754203    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.754996 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.754977    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.755091 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.755008    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.755091 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.755025    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.755091 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.755060    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.765056 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.765037    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.765142 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.765059    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-226.ec2.internal\": node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:11.797002 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.796936    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:11.818076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.818055    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"]
Sep 16 18:45:11.818159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.818114    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.819710 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.819692    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.819794 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.819716    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.819794 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.819727    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.821875 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.821862    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.821924 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.821912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.821958 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.821943    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.822565 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822551    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.822626 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822578    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.822626 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822588    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.822692 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822552    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.822692 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822657    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.822692 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.822668    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.824765 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.824746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.824875 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.824770    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:11.825443 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.825426    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:11.825516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.825455    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:11.825516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.825467    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:11.843445 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.843426    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-226.ec2.internal\" not found" node="ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.847837 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.847808    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-226.ec2.internal\" not found" node="ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.897161 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.897140    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:11.909064 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.909045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.909139 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.909067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.909139 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:11.909084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bf20f76c0d057976dff6a847f7d0954-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"5bf20f76c0d057976dff6a847f7d0954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:11.997333 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:11.997310    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.009672 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.009726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.009726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bf20f76c0d057976dff6a847f7d0954-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"5bf20f76c0d057976dff6a847f7d0954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.009792 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.009792 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a050e3969e6ff2aa8f7bbb45edf60e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"6a050e3969e6ff2aa8f7bbb45edf60e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.009792 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.009768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bf20f76c0d057976dff6a847f7d0954-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"5bf20f76c0d057976dff6a847f7d0954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.098130 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.098068    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.146553 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.146535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.151312 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.151225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.198291 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.198261    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.298768 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.298737    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.399934 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.399912    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.500389 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.500361    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.539909 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.539881    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:12.540248 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.540002    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:12.601287 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:12.601262    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found"
Sep 16 18:45:12.623317 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.623292    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:12.645287 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.645229    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:12 +0000 UTC" deadline="2028-04-20 12:43:52.439994765 +0000 UTC"
Sep 16 18:45:12.645287 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.645255    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13961h58m39.794742891s"
Sep 16 18:45:12.693805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.693785    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:12.708146 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.708127    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.729070 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.729053    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:12.730680 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.730662    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal"
Sep 16 18:45:12.774932 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.774910    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:12.790422 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:12.790390    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a050e3969e6ff2aa8f7bbb45edf60e2.slice/crio-951feb2d8ea5871b1fdb68b58b4788dfa559b7634f2fb3eec2acddcac30a5058 WatchSource:0}: Error finding container 951feb2d8ea5871b1fdb68b58b4788dfa559b7634f2fb3eec2acddcac30a5058: Status 404 returned error can't find the container with id 951feb2d8ea5871b1fdb68b58b4788dfa559b7634f2fb3eec2acddcac30a5058
Sep 16 18:45:12.800696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:12.800680    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:12.806558 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:12.806536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf20f76c0d057976dff6a847f7d0954.slice/crio-860e47588c519695608f5911fa14f1540025c12699ec7308959e25f1a69309ce WatchSource:0}: Error finding container 860e47588c519695608f5911fa14f1540025c12699ec7308959e25f1a69309ce: Status 404 returned error can't find the container with id 860e47588c519695608f5911fa14f1540025c12699ec7308959e25f1a69309ce
Sep 16 18:45:13.032089 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.032026    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:13.585430 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.585384    2563 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:13.596981 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.596959    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:13.597349 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.597320    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-77tmj","kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hl2hk","openshift-multus/multus-w6zm4","openshift-network-diagnostics/network-check-target-jmbzj","openshift-network-operator/iptables-alerter-xtrmf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc","openshift-dns/node-resolver-s8xph","openshift-image-registry/node-ca-mq86t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-vznp7","openshift-multus/network-metrics-daemon-mmtgl","openshift-ovn-kubernetes/ovnkube-node-xdr6v","kube-system/global-pull-secret-syncer-57jlt"]
Sep 16 18:45:13.600061 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.600042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.602538 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.602517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.604258 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.604221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8dqkm\""
Sep 16 18:45:13.604782 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.604756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.604881 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.604761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.605066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.605048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.605829 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.605806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:13.606635 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.606618    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:13.607912 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.607874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.609707 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.609685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:13.609804 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.609762    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:13.610566 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.610493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:13.610651 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.610584    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.610651 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.610589    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.610651 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.610596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.610781 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.610742    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.612139 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:13.612262 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.612411 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:13.612473 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tjbx\""
Sep 16 18:45:13.612583 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:13.612583 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.612579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kpj5f\""
Sep 16 18:45:13.614668 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.614643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.617363 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.617342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.618425 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.618387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.618425 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.618405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:13.618574 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.618544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-75hml\""
Sep 16 18:45:13.618738 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.618647    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:13.619066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-modprobe-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-tmp\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-k8s-cni-cncf-io\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-daemon-config\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-sys-fs\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cnibin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-hostroot\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-script-lib\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cni-binary-copy\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-socket-dir-parent\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-ovn\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-netd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-conf\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-etc-tuned\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-device-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-netns\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-systemd-units\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-lib-modules\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.619605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6vg\" (UniqueName: \"kubernetes.io/projected/1c58745a-9e5c-439c-825b-fe2e03450003-kube-api-access-4n6vg\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcxj\" (UniqueName: \"kubernetes.io/projected/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kube-api-access-grcxj\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-kubernetes\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-etc-kubernetes\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-socket-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-kubelet\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-var-lib-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-etc-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.620153    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.619824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-bin\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-systemd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-log-socket\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-run\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-sys\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.620457 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-system-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.620848 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-registration-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.620848 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-var-lib-kubelet\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.620848 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-multus\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.620848 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-multus-certs\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.621022 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.620951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.621022 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-slash\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621526 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-netns\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621627 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5hz\" (UniqueName: \"kubernetes.io/projected/ae3c2102-ddc8-4558-b65c-77ec51194fcf-kube-api-access-5j5hz\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621627 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cj7\" (UniqueName: \"kubernetes.io/projected/9c0aa788-3d2f-4abb-9e00-72cb1181215f-kube-api-access-j5cj7\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.621627 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-node-log\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621627 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-host\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-bin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-conf-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-config\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-env-overrides\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.621762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysconfig\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.622087 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-systemd\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.622087 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-kubelet\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.622087 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.621924    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.623533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.623633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.624343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.624399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.624641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.624690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzwtt\""
Sep 16 18:45:13.625373 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.624642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.625789 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.625584    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.625789 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.625600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pg8wv\""
Sep 16 18:45:13.625883 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.625873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc8ch\""
Sep 16 18:45:13.626037 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.626026    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.626863 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.626215    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:13.626863 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.626227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.626863 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.626842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:13.629024 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.629004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.629123 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.629075    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:13.637506 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.637483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:13.638566 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.638547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:13.639712 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.639694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hdqkd\""
Sep 16 18:45:13.639910 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.639878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:13.640159 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.640137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:13.640961 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.640943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-95nbs\""
Sep 16 18:45:13.649066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.649042    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:13.659504 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.659482    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvlg8"
Sep 16 18:45:13.676643 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.676624    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvlg8"
Sep 16 18:45:13.708632 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.708603    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:13.722049 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.721981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-etc-kubernetes\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722049 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-socket-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.722049 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-kubelet\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-var-lib-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-etc-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-kubelet\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-bin\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-var-lib-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-etc-openvswitch\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-systemd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-log-socket\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-run\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-sys\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-system-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-etc-kubernetes\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-kubelet-config\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-registration-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-var-lib-kubelet\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-multus\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-socket-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-multus-certs\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-multus-certs\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa9e397-802f-4959-8eed-d23fa20e26b7-iptables-alerter-script\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1224ee06-367b-4d82-a884-2ad98cfaed41-agent-certs\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-log-socket\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.722825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vst\" (UniqueName: \"kubernetes.io/projected/14610c4a-a5e4-4bbc-a969-333868322d1c-kube-api-access-m8vst\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-bin\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-slash\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-systemd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-netns\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5hz\" (UniqueName: \"kubernetes.io/projected/ae3c2102-ddc8-4558-b65c-77ec51194fcf-kube-api-access-5j5hz\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-registration-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cj7\" (UniqueName: \"kubernetes.io/projected/9c0aa788-3d2f-4abb-9e00-72cb1181215f-kube-api-access-j5cj7\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-run\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0672e4fc-19cc-464f-9123-8b90a9ce0107-host\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-sys\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0672e4fc-19cc-464f-9123-8b90a9ce0107-serviceca\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-dbus\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.722995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-netns\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-slash\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-system-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.723618 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-var-lib-kubelet\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-multus\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-system-cni-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-node-log\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-host\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-bin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-conf-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-config\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-host\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-conf-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-cni-bin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-node-log\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxcd\" (UniqueName: \"kubernetes.io/projected/0672e4fc-19cc-464f-9123-8b90a9ce0107-kube-api-access-2hxcd\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1224ee06-367b-4d82-a884-2ad98cfaed41-konnectivity-ca\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14610c4a-a5e4-4bbc-a969-333868322d1c-hosts-file\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.724530 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-env-overrides\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysconfig\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" event={"ID":"5bf20f76c0d057976dff6a847f7d0954","Type":"ContainerStarted","Data":"860e47588c519695608f5911fa14f1540025c12699ec7308959e25f1a69309ce"}
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-systemd\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-kubelet\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkhx\" (UniqueName: \"kubernetes.io/projected/2aa9e397-802f-4959-8eed-d23fa20e26b7-kube-api-access-8mkhx\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmbf\" (UniqueName: \"kubernetes.io/projected/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-kube-api-access-9qmbf\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-var-lib-kubelet\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-modprobe-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-tmp\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-k8s-cni-cncf-io\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-daemon-config\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-env-overrides\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.723989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14610c4a-a5e4-4bbc-a969-333868322d1c-tmp-dir\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.725391 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-modprobe-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-config\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-sys-fs\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysconfig\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-systemd\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cnibin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-hostroot\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-sys-fs\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-script-lib\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-hostroot\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cni-binary-copy\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-socket-dir-parent\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-k8s-cni-cncf-io\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa9e397-802f-4959-8eed-d23fa20e26b7-host-slash\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.726270 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-ovn\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-netd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-conf\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-etc-tuned\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-socket-dir-parent\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cnibin\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-device-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724732    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-daemon-config\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-netns\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/653badd4-56f2-45a9-b73a-55b1ae51b5d2-device-dir\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-cni-netd\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovnkube-script-lib\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-run-ovn\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cnibin\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.727149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-host-run-netns\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-conf\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.724999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0aa788-3d2f-4abb-9e00-72cb1181215f-cni-binary-copy\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktxp\" (UniqueName: \"kubernetes.io/projected/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-kube-api-access-xktxp\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-systemd-units\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae3c2102-ddc8-4558-b65c-77ec51194fcf-systemd-units\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-lib-modules\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6vg\" (UniqueName: \"kubernetes.io/projected/1c58745a-9e5c-439c-825b-fe2e03450003-kube-api-access-4n6vg\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcxj\" (UniqueName: \"kubernetes.io/projected/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kube-api-access-grcxj\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-kubernetes\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-sysctl-d\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-lib-modules\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0aa788-3d2f-4abb-9e00-72cb1181215f-multus-cni-dir\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c58745a-9e5c-439c-825b-fe2e03450003-etc-kubernetes\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.728078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.725681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"6a050e3969e6ff2aa8f7bbb45edf60e2","Type":"ContainerStarted","Data":"951feb2d8ea5871b1fdb68b58b4788dfa559b7634f2fb3eec2acddcac30a5058"}
Sep 16 18:45:13.729576 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.729551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-tmp\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.729701 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.729683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae3c2102-ddc8-4558-b65c-77ec51194fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.730018 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.729994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c58745a-9e5c-439c-825b-fe2e03450003-etc-tuned\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.739634 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.739574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cj7\" (UniqueName: \"kubernetes.io/projected/9c0aa788-3d2f-4abb-9e00-72cb1181215f-kube-api-access-j5cj7\") pod \"multus-w6zm4\" (UID: \"9c0aa788-3d2f-4abb-9e00-72cb1181215f\") " pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.739909 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.739874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcxj\" (UniqueName: \"kubernetes.io/projected/653badd4-56f2-45a9-b73a-55b1ae51b5d2-kube-api-access-grcxj\") pod \"aws-ebs-csi-driver-node-z2fpc\" (UID: \"653badd4-56f2-45a9-b73a-55b1ae51b5d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.741220 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.741196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6vg\" (UniqueName: \"kubernetes.io/projected/1c58745a-9e5c-439c-825b-fe2e03450003-kube-api-access-4n6vg\") pod \"tuned-hl2hk\" (UID: \"1c58745a-9e5c-439c-825b-fe2e03450003\") " pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.742424 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.742401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5hz\" (UniqueName: \"kubernetes.io/projected/ae3c2102-ddc8-4558-b65c-77ec51194fcf-kube-api-access-5j5hz\") pod \"ovnkube-node-xdr6v\" (UID: \"ae3c2102-ddc8-4558-b65c-77ec51194fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.826479 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14610c4a-a5e4-4bbc-a969-333868322d1c-tmp-dir\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.826479 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa9e397-802f-4959-8eed-d23fa20e26b7-host-slash\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cnibin\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktxp\" (UniqueName: \"kubernetes.io/projected/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-kube-api-access-xktxp\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.826721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-kubelet-config\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa9e397-802f-4959-8eed-d23fa20e26b7-iptables-alerter-script\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1224ee06-367b-4d82-a884-2ad98cfaed41-agent-certs\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vst\" (UniqueName: \"kubernetes.io/projected/14610c4a-a5e4-4bbc-a969-333868322d1c-kube-api-access-m8vst\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14610c4a-a5e4-4bbc-a969-333868322d1c-tmp-dir\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0672e4fc-19cc-464f-9123-8b90a9ce0107-host\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.826858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0672e4fc-19cc-464f-9123-8b90a9ce0107-serviceca\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-dbus\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.827168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cnibin\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa9e397-802f-4959-8eed-d23fa20e26b7-host-slash\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0672e4fc-19cc-464f-9123-8b90a9ce0107-serviceca\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-dbus\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-system-cni-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-system-cni-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827531 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-kubelet-config\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:13.827825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-cni-binary-copy\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0672e4fc-19cc-464f-9123-8b90a9ce0107-host\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.827825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.827719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.827970 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.827870    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:13.827970 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.827938    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.327917523 +0000 UTC m=+3.066493020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:13.828117 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.828231 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxcd\" (UniqueName: \"kubernetes.io/projected/0672e4fc-19cc-464f-9123-8b90a9ce0107-kube-api-access-2hxcd\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.828309 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1224ee06-367b-4d82-a884-2ad98cfaed41-konnectivity-ca\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.828309 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14610c4a-a5e4-4bbc-a969-333868322d1c-hosts-file\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.828411 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.828411 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkhx\" (UniqueName: \"kubernetes.io/projected/2aa9e397-802f-4959-8eed-d23fa20e26b7-kube-api-access-8mkhx\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.828411 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmbf\" (UniqueName: \"kubernetes.io/projected/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-kube-api-access-9qmbf\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.828566 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14610c4a-a5e4-4bbc-a969-333868322d1c-hosts-file\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.828566 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa9e397-802f-4959-8eed-d23fa20e26b7-iptables-alerter-script\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.828566 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.828499    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:13.828566 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.828545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.328528801 +0000 UTC m=+3.067104294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:13.828861 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.828821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1224ee06-367b-4d82-a884-2ad98cfaed41-konnectivity-ca\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.830863 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.830835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1224ee06-367b-4d82-a884-2ad98cfaed41-agent-certs\") pod \"konnectivity-agent-77tmj\" (UID: \"1224ee06-367b-4d82-a884-2ad98cfaed41\") " pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.849339 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.849221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxcd\" (UniqueName: \"kubernetes.io/projected/0672e4fc-19cc-464f-9123-8b90a9ce0107-kube-api-access-2hxcd\") pod \"node-ca-mq86t\" (UID: \"0672e4fc-19cc-464f-9123-8b90a9ce0107\") " pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.849576 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.849493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktxp\" (UniqueName: \"kubernetes.io/projected/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-kube-api-access-xktxp\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:13.851414 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.851381    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:13.851414 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.851409    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:13.851595 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.851424    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:13.851595 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:13.851492    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:14.35147617 +0000 UTC m=+3.090051669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:13.853967 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.853941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmbf\" (UniqueName: \"kubernetes.io/projected/2c6d9c30-89bc-4ca4-8202-f77f3871e8ae-kube-api-access-9qmbf\") pod \"multus-additional-cni-plugins-vznp7\" (UID: \"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae\") " pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:13.855127 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.855105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vst\" (UniqueName: \"kubernetes.io/projected/14610c4a-a5e4-4bbc-a969-333868322d1c-kube-api-access-m8vst\") pod \"node-resolver-s8xph\" (UID: \"14610c4a-a5e4-4bbc-a969-333868322d1c\") " pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.856676 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.856654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkhx\" (UniqueName: \"kubernetes.io/projected/2aa9e397-802f-4959-8eed-d23fa20e26b7-kube-api-access-8mkhx\") pod \"iptables-alerter-xtrmf\" (UID: \"2aa9e397-802f-4959-8eed-d23fa20e26b7\") " pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.914725 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.914701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc"
Sep 16 18:45:13.923562 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.923539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:13.932460 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.932304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hl2hk"
Sep 16 18:45:13.938301 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.938279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w6zm4"
Sep 16 18:45:13.948247 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.948224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtrmf"
Sep 16 18:45:13.953074 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.953052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mq86t"
Sep 16 18:45:13.959955 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.959863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:13.966893 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.966871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s8xph"
Sep 16 18:45:13.973022 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:13.972992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vznp7"
Sep 16 18:45:14.208104 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:14.207879    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3c2102_ddc8_4558_b65c_77ec51194fcf.slice/crio-651f9c96d0e6e1821005ed8648c8adf30389ccc4802c17ded5408ebe1d84ece7 WatchSource:0}: Error finding container 651f9c96d0e6e1821005ed8648c8adf30389ccc4802c17ded5408ebe1d84ece7: Status 404 returned error can't find the container with id 651f9c96d0e6e1821005ed8648c8adf30389ccc4802c17ded5408ebe1d84ece7
Sep 16 18:45:14.211679 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:14.211656    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c58745a_9e5c_439c_825b_fe2e03450003.slice/crio-58672ec015fc68458c07e203c091fde16366f1a788595914199c2795fbe01fb7 WatchSource:0}: Error finding container 58672ec015fc68458c07e203c091fde16366f1a788595914199c2795fbe01fb7: Status 404 returned error can't find the container with id 58672ec015fc68458c07e203c091fde16366f1a788595914199c2795fbe01fb7
Sep 16 18:45:14.214470 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:14.214418    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653badd4_56f2_45a9_b73a_55b1ae51b5d2.slice/crio-8152d4f47ac763510af71a88934b64a08dc4f76d22d9face658ff1247826fe97 WatchSource:0}: Error finding container 8152d4f47ac763510af71a88934b64a08dc4f76d22d9face658ff1247826fe97: Status 404 returned error can't find the container with id 8152d4f47ac763510af71a88934b64a08dc4f76d22d9face658ff1247826fe97
Sep 16 18:45:14.243853 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:14.243826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6d9c30_89bc_4ca4_8202_f77f3871e8ae.slice/crio-193e84a020e02d7eebb8a772bf6171acb6fe5322a4ffe24b9397c0fd5f3f1e02 WatchSource:0}: Error finding container 193e84a020e02d7eebb8a772bf6171acb6fe5322a4ffe24b9397c0fd5f3f1e02: Status 404 returned error can't find the container with id 193e84a020e02d7eebb8a772bf6171acb6fe5322a4ffe24b9397c0fd5f3f1e02
Sep 16 18:45:14.246615 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:14.246591    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0672e4fc_19cc_464f_9123_8b90a9ce0107.slice/crio-8572069f071dd7d60d2f661a94354cd1d46f37f431b1c557839bf81e9cbb436b WatchSource:0}: Error finding container 8572069f071dd7d60d2f661a94354cd1d46f37f431b1c557839bf81e9cbb436b: Status 404 returned error can't find the container with id 8572069f071dd7d60d2f661a94354cd1d46f37f431b1c557839bf81e9cbb436b
Sep 16 18:45:14.332974 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.332931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:14.333095 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.333010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:14.333151 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.333098    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:14.333151 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.333127    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:14.333226 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.333197    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.33314967 +0000 UTC m=+4.071725153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:14.333226 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.333218    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.333206867 +0000 UTC m=+4.071782351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:14.433838 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.433794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:14.434005 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.433976    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:14.434005 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.434004    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:14.434109 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.434019    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:14.434109 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:14.434081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.43406131 +0000 UTC m=+4.172636802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:14.680452 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.680413    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:14 +0000 UTC" deadline="2028-05-15 09:26:49.747603157 +0000 UTC"
Sep 16 18:45:14.680452 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.680445    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14558h41m35.067161796s"
Sep 16 18:45:14.728463 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.728425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc" event={"ID":"653badd4-56f2-45a9-b73a-55b1ae51b5d2","Type":"ContainerStarted","Data":"8152d4f47ac763510af71a88934b64a08dc4f76d22d9face658ff1247826fe97"}
Sep 16 18:45:14.730005 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.729973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"651f9c96d0e6e1821005ed8648c8adf30389ccc4802c17ded5408ebe1d84ece7"}
Sep 16 18:45:14.734849 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.734813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"6a050e3969e6ff2aa8f7bbb45edf60e2","Type":"ContainerStarted","Data":"ffaab7bf771d618a5fd4408827ef0f6eb6d839344ace9fc50d2fd3839c9eae3b"}
Sep 16 18:45:14.736144 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.736120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerStarted","Data":"193e84a020e02d7eebb8a772bf6171acb6fe5322a4ffe24b9397c0fd5f3f1e02"}
Sep 16 18:45:14.737260 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.737240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-77tmj" event={"ID":"1224ee06-367b-4d82-a884-2ad98cfaed41","Type":"ContainerStarted","Data":"dbfce445e8a1ef2ceff54e0941a9628c788f75bc904062662e1982c64245abff"}
Sep 16 18:45:14.738371 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.738314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtrmf" event={"ID":"2aa9e397-802f-4959-8eed-d23fa20e26b7","Type":"ContainerStarted","Data":"900279d54f14c81f532f8353976786a89dc5f943e0c84fd83f7ac85b8b027527"}
Sep 16 18:45:14.739536 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.739512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w6zm4" event={"ID":"9c0aa788-3d2f-4abb-9e00-72cb1181215f","Type":"ContainerStarted","Data":"9f363030057635b3df40c02615aea09ad537e302748e78d0bdfb7febbff0a81f"}
Sep 16 18:45:14.740761 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.740731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hl2hk" event={"ID":"1c58745a-9e5c-439c-825b-fe2e03450003","Type":"ContainerStarted","Data":"58672ec015fc68458c07e203c091fde16366f1a788595914199c2795fbe01fb7"}
Sep 16 18:45:14.742006 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.741985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s8xph" event={"ID":"14610c4a-a5e4-4bbc-a969-333868322d1c","Type":"ContainerStarted","Data":"be666f760de7c9be70aa166995b6e601cffb2e6098dbbc28da157900b5572d69"}
Sep 16 18:45:14.743163 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:14.743136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mq86t" event={"ID":"0672e4fc-19cc-464f-9123-8b90a9ce0107","Type":"ContainerStarted","Data":"8572069f071dd7d60d2f661a94354cd1d46f37f431b1c557839bf81e9cbb436b"}
Sep 16 18:45:15.341282 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.341243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:15.341458 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.341303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:15.341458 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.341417    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:15.341611 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.341469    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.341455362 +0000 UTC m=+6.080030854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:15.341611 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.341514    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:15.341611 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.341536    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.341529491 +0000 UTC m=+6.080104982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:15.441623 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.441583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:15.441838 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.441821    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:15.441927 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.441846    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:15.441927 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.441879    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:15.442043 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.441939    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:17.441921027 +0000 UTC m=+6.180496523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:15.680919 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.680743    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:14 +0000 UTC" deadline="2028-07-01 22:41:52.049471476 +0000 UTC"
Sep 16 18:45:15.680919 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.680807    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15699h56m36.368668616s"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.722215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.722351    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.722773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.722862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.722951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:15.723155 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:15.723041    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:15.759181 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.759145    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a050e3969e6ff2aa8f7bbb45edf60e2" containerID="ffaab7bf771d618a5fd4408827ef0f6eb6d839344ace9fc50d2fd3839c9eae3b" exitCode=0
Sep 16 18:45:15.759344 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:15.759203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"6a050e3969e6ff2aa8f7bbb45edf60e2","Type":"ContainerDied","Data":"ffaab7bf771d618a5fd4408827ef0f6eb6d839344ace9fc50d2fd3839c9eae3b"}
Sep 16 18:45:16.772446 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:16.771383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" event={"ID":"5bf20f76c0d057976dff6a847f7d0954","Type":"ContainerStarted","Data":"ccacc79f9f3f865d41a467ebb469333bf6164128b7ed9b9860d912f22d110b43"}
Sep 16 18:45:16.780150 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:16.779659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"6a050e3969e6ff2aa8f7bbb45edf60e2","Type":"ContainerStarted","Data":"522ea3b7ff92ea87a7aa8629ac5d0ffd65be25051eade0195ca34f3c6b69258f"}
Sep 16 18:45:16.795273 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:16.795120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" podStartSLOduration=3.7950991800000002 podStartE2EDuration="3.79509918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:16.794034984 +0000 UTC m=+5.532610489" watchObservedRunningTime="2026-09-16 18:45:16.79509918 +0000 UTC m=+5.533674684"
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.358538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.358624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.358769    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.358929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.358907887 +0000 UTC m=+10.097483379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.359384    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:17.359496 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.359461    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.359444737 +0000 UTC m=+10.098020233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:17.460089 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.459427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:17.460089 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.459630    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:17.460089 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.459655    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:17.460089 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.459671    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:17.460089 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.459728    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.459708321 +0000 UTC m=+10.198283813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:17.719866 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.718992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:17.719866 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.719139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:17.719866 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.719572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:17.719866 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.719671    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:17.720501 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:17.720290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:17.720501 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:17.720393    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:19.720305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:19.720430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:19.720799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:19.720912    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:19.720976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:19.721120 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:19.721062    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:21.393291 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.393252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:21.393802 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.393309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:21.393802 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.393414    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:21.393802 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.393480    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.393459347 +0000 UTC m=+18.132034853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:21.393802 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.393672    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:21.393802 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.393728    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.39371194 +0000 UTC m=+18.132287433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:21.494949 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.494365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:21.494949 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.494523    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:21.494949 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.494545    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:21.494949 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.494558    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:21.494949 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.494621    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.494602251 +0000 UTC m=+18.233177756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:21.719742 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.719488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:21.720091 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.720058    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:21.720562 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.720542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:21.720805 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.720772    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:21.720964 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:21.720857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:21.721182 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:21.721161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:23.717937 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:23.717903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:23.718358 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:23.718039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:23.718358 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:23.718102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:23.718358 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:23.718131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:23.718358 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:23.718224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:23.718358 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:23.718297    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:25.718481 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:25.718456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:25.718852 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:25.718490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:25.718852 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:25.718453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:25.718852 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:25.718580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:25.718852 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:25.718664    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:25.718852 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:25.718739    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:27.721048 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:27.721013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:27.721495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:27.721157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:27.721495 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:27.721168    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:27.721495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:27.721192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:27.721495 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:27.721282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:27.721495 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:27.721363    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:29.453019 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.452986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:29.453449 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.453047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:29.453449 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.453146    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:29.453449 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.453206    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.453191198 +0000 UTC m=+34.191766683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:29.453449 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.453147    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:29.453449 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.453302    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.453283312 +0000 UTC m=+34.191858811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:29.553895 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.553865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:29.554060 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.554041    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:29.554141 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.554067    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:29.554141 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.554080    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:29.554246 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.554141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.554123387 +0000 UTC m=+34.292698878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:29.720626 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.720602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:29.720763 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.720720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:29.720836 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.720767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:29.720892 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.720854    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:29.720892 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:29.720890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:29.720984 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:29.720951    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:31.718212 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:31.718181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:31.718710 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:31.718291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:31.718710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:31.718286    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:31.718710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:31.718372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:31.718710 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:31.718385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:31.718710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:31.718510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:33.718016 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.717990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:33.718466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.718030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:33.718466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.717988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:33.718466 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:33.718111    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:33.718466 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:33.718194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:33.718466 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:33.718325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:33.819896 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.819867    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="ec24208d95655b50ba3a3871434f6e04a96c1ecf7f0dfe56a1f93a0ffcc5382e" exitCode=0
Sep 16 18:45:33.820047 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.819945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"ec24208d95655b50ba3a3871434f6e04a96c1ecf7f0dfe56a1f93a0ffcc5382e"}
Sep 16 18:45:33.821608 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.821581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-77tmj" event={"ID":"1224ee06-367b-4d82-a884-2ad98cfaed41","Type":"ContainerStarted","Data":"9a0807945270bc58fd5254285981dd81e55d8d59a5e871b92bc040775babdc15"}
Sep 16 18:45:33.823484 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.823461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w6zm4" event={"ID":"9c0aa788-3d2f-4abb-9e00-72cb1181215f","Type":"ContainerStarted","Data":"97d4a26f65a36559701bda3d6c82e3d1c9dd26d2a1fdfffce952d2212db9fc86"}
Sep 16 18:45:33.825195 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.825170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hl2hk" event={"ID":"1c58745a-9e5c-439c-825b-fe2e03450003","Type":"ContainerStarted","Data":"4d88b1b603b09362197236705833f04dbc4413f12ef96d6d4440483bc1b36ed1"}
Sep 16 18:45:33.828089 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.828066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s8xph" event={"ID":"14610c4a-a5e4-4bbc-a969-333868322d1c","Type":"ContainerStarted","Data":"2c720fea83c24ca3e11b6335d940b00111ec471d51634b22d511f3b7c989f52b"}
Sep 16 18:45:33.829706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.829677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mq86t" event={"ID":"0672e4fc-19cc-464f-9123-8b90a9ce0107","Type":"ContainerStarted","Data":"d20c939f39c6e12f13bcd94b002b15a9967233ca810eafe1e56ca64272cfd7d4"}
Sep 16 18:45:33.831262 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.831237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc" event={"ID":"653badd4-56f2-45a9-b73a-55b1ae51b5d2","Type":"ContainerStarted","Data":"a3a4d8b070115ecfb40e9c5685c728b13e993bbd77735b5e7793682e621cf5a6"}
Sep 16 18:45:33.834537 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"47919591dd9f477643218a746043c72fd053f559c360f41e093ce7189a870fd0"}
Sep 16 18:45:33.834631 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"6eb808f757ffac1905211f768e5bc9d16a3df926099660adf95850f2b6549267"}
Sep 16 18:45:33.834685 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"cdb6ccba983941457c0b524e1f4e1615b8aeb701c9d78817b440f4c43e20d0b8"}
Sep 16 18:45:33.834723 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"ec20827847ebc02a30f8efceeaebe0ccfa0f9e8b892c8b9c84a03fc3b2f48a32"}
Sep 16 18:45:33.834757 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"a2785e3465b1d831b05753e15f39b2af6f98b463be1603a847308e171eb0ba58"}
Sep 16 18:45:33.834791 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.834781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"bb4f3cbc781addf3c88cce94961788159c81281e8d19825d474706d88796d88b"}
Sep 16 18:45:33.846141 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.846094    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" podStartSLOduration=20.846082006 podStartE2EDuration="20.846082006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:16.816217305 +0000 UTC m=+5.554792808" watchObservedRunningTime="2026-09-16 18:45:33.846082006 +0000 UTC m=+22.584657504"
Sep 16 18:45:33.885254 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.885213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mq86t" podStartSLOduration=3.249465843 podStartE2EDuration="21.885196684s" totalImagesPullingTime="18.635730841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:33.864503952 +0000 UTC m=+22.603079449" watchObservedRunningTime="2026-09-16 18:45:33.885196684 +0000 UTC m=+22.623772188"
Sep 16 18:45:33.909206 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.909164    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-77tmj" podStartSLOduration=3.243331205 podStartE2EDuration="21.909150772s" totalImagesPullingTime="18.665819567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:33.885844505 +0000 UTC m=+22.624420007" watchObservedRunningTime="2026-09-16 18:45:33.909150772 +0000 UTC m=+22.647726270"
Sep 16 18:45:33.909313 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.909261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hl2hk" podStartSLOduration=3.20691899 podStartE2EDuration="21.90925526s" totalImagesPullingTime="18.70233627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:33.90865748 +0000 UTC m=+22.647232978" watchObservedRunningTime="2026-09-16 18:45:33.90925526 +0000 UTC m=+22.647830760"
Sep 16 18:45:33.934455 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:33.934418    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w6zm4" podStartSLOduration=3.24428108 podStartE2EDuration="21.934405955s" totalImagesPullingTime="18.690124875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:33.934129146 +0000 UTC m=+22.672704644" watchObservedRunningTime="2026-09-16 18:45:33.934405955 +0000 UTC m=+22.672981459"
Sep 16 18:45:34.004879 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.004804    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s8xph" podStartSLOduration=3.330568231 podStartE2EDuration="22.004788777s" totalImagesPullingTime="18.674220546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.004382285 +0000 UTC m=+22.742957783" watchObservedRunningTime="2026-09-16 18:45:34.004788777 +0000 UTC m=+22.743364272"
Sep 16 18:45:34.386512 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.386341    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:34.404300 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.404279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:34.404726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.404708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:34.665403 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.665251    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:34.386511244Z","UUID":"a3cb884d-fdba-46e1-ae5a-be6090cb2942","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:34.666674 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.666658    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:34.666792 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.666686    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:34.838723 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.838676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtrmf" event={"ID":"2aa9e397-802f-4959-8eed-d23fa20e26b7","Type":"ContainerStarted","Data":"3122619a7d0ea59bc6cf2a2c29dc65bed6954052f61ede31a707f64e234d32e4"}
Sep 16 18:45:34.841357 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.841316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc" event={"ID":"653badd4-56f2-45a9-b73a-55b1ae51b5d2","Type":"ContainerStarted","Data":"bea487cc0e4606bb03a2524a95ae261bf018643edec5f3441fa58ff9fe144840"}
Sep 16 18:45:34.842004 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.841868    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:34.842301 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.842279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-77tmj"
Sep 16 18:45:34.861754 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:34.861702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xtrmf" podStartSLOduration=4.219591319 podStartE2EDuration="22.86168604s" totalImagesPullingTime="18.642094721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.861432242 +0000 UTC m=+23.600007740" watchObservedRunningTime="2026-09-16 18:45:34.86168604 +0000 UTC m=+23.600261539"
Sep 16 18:45:35.718514 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:35.718438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:35.718703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:35.718446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:35.718703 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:35.718537    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:35.718703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:35.718569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:35.718703 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:35.718659    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:35.718912 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:35.718799    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:35.845991 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:35.845959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc" event={"ID":"653badd4-56f2-45a9-b73a-55b1ae51b5d2","Type":"ContainerStarted","Data":"021367ba00226aae1addc493aaae637ee3df633acc68b7f6269448f9563dd6d4"}
Sep 16 18:45:35.871274 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:35.871229    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z2fpc" podStartSLOduration=3.050054269 podStartE2EDuration="23.871212983s" totalImagesPullingTime="20.821158714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.870729763 +0000 UTC m=+24.609305262" watchObservedRunningTime="2026-09-16 18:45:35.871212983 +0000 UTC m=+24.609788481"
Sep 16 18:45:36.851723 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:36.851678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"b41df92fbe564a27dccfa8e7d3bec63cc4ba730e50f5052a3dc6ef476e97a228"}
Sep 16 18:45:37.717948 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:37.717912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:37.717948 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:37.717936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:37.718236 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:37.718029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:37.718236 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:37.718104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:37.718236 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:37.718166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:37.718381 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:37.718282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:38.858093 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.857888    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="56f1479ab67e812afc8b1fe6d6a5597aa72e6477fcbbba57c002a8d576b35ed1" exitCode=0
Sep 16 18:45:38.858630 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.857973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"56f1479ab67e812afc8b1fe6d6a5597aa72e6477fcbbba57c002a8d576b35ed1"}
Sep 16 18:45:38.862093 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.862068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" event={"ID":"ae3c2102-ddc8-4558-b65c-77ec51194fcf","Type":"ContainerStarted","Data":"7e725da1a96155529552cc7aaa052c7fb67adf5e11bce7681720751346521901"}
Sep 16 18:45:38.862703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.862688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:38.862784 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.862774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:38.877188 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.877169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:38.877322 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:38.877288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:39.717754 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.717722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:39.717937 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.717864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:39.718011 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.717871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:39.718064 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.717999    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:39.718064 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.718019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:39.718164 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.718136    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:39.864451 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.864424    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:39.975136 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.975053    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v" podStartSLOduration=9.086963775 podStartE2EDuration="27.975033205s" totalImagesPullingTime="18.88806943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:38.93444576 +0000 UTC m=+27.673021257" watchObservedRunningTime="2026-09-16 18:45:39.975033205 +0000 UTC m=+28.713608704"
Sep 16 18:45:39.976513 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.976490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mmtgl"]
Sep 16 18:45:39.976668 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.976614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:39.976751 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.976728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:39.982294 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.982271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57jlt"]
Sep 16 18:45:39.982363 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.982355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:39.982447 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.982428    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:39.987571 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.987551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmbzj"]
Sep 16 18:45:39.987664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:39.987626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:39.987712 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:39.987700    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:40.868776 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:40.868739    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="ac5a4047835213034e1b3548f117cf4001d7b2f064723f7d57063f4b9c7abd3b" exitCode=0
Sep 16 18:45:40.869233 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:40.868833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"ac5a4047835213034e1b3548f117cf4001d7b2f064723f7d57063f4b9c7abd3b"}
Sep 16 18:45:40.869233 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:40.868917    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:41.072548 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:41.072522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:45:41.718775 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:41.718632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:41.718950 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:41.718693    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:41.718950 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:41.718842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:41.719041 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:41.718941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:41.719041 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:41.718712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:41.719123 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:41.719042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:42.875445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:42.875414    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="8db29e3f05bb480359449c2adcf68bd3d6db719d017ed677191fb1e6fc079c5b" exitCode=0
Sep 16 18:45:42.875770 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:42.875458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"8db29e3f05bb480359449c2adcf68bd3d6db719d017ed677191fb1e6fc079c5b"}
Sep 16 18:45:43.717654 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:43.717624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:43.717861 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:43.717624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:43.717861 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:43.717728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:43.717981 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:43.717859    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:43.717981 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:43.717624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:43.717981 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:43.717973    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:45.477719 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.477687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:45.478171 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.477751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:45.478171 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.477862    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.478171 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.477913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.477893295 +0000 UTC m=+66.216468771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.478171 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.477860    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.478171 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.478008    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret podName:c3aa8d45-86ee-44c6-b9dc-01e878c909b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.477972096 +0000 UTC m=+66.216547587 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret") pod "global-pull-secret-syncer-57jlt" (UID: "c3aa8d45-86ee-44c6-b9dc-01e878c909b5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.578552 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.578526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:45.578734 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.578710    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:45.578880 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.578742    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:45.578880 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.578756    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mv5pr for pod openshift-network-diagnostics/network-check-target-jmbzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.578880 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.578836    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr podName:f9afdf29-14a2-48be-8732-7981a9c78d80 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.578801611 +0000 UTC m=+66.317377103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mv5pr" (UniqueName: "kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr") pod "network-check-target-jmbzj" (UID: "f9afdf29-14a2-48be-8732-7981a9c78d80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.717989 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.717953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:45.717989 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.717987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:45.718218 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:45.717953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:45.718218 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.718094    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:45:45.718218 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.718174    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmbzj" podUID="f9afdf29-14a2-48be-8732-7981a9c78d80"
Sep 16 18:45:45.718371 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:45.718264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57jlt" podUID="c3aa8d45-86ee-44c6-b9dc-01e878c909b5"
Sep 16 18:45:46.075200 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.075173    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeReady"
Sep 16 18:45:46.075364 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.075299    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:46.115183 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.115156    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:45:46.117644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.117622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.119033 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.119005    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"]
Sep 16 18:45:46.120694 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.120670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:46.120801 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.120718    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"]
Sep 16 18:45:46.120989 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.120959    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:46.120989 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.120977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:46.121145 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.121046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.121516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.121498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s9n6t\""
Sep 16 18:45:46.122503 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.122420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"]
Sep 16 18:45:46.122597 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.122543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.124689 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.124331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.125884 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.125761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:45:46.125884 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.125867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:45:46.126040 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.125946    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dwd9j\""
Sep 16 18:45:46.126040 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.125987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:45:46.126135 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.126086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:45:46.127429 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.127411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:45:46.129002 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.128976    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:45:46.129097 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.129034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:45:46.129162 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.129125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:45:46.129219 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.129158    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:46.129274 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.129247    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:45:46.133076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.133055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"]
Sep 16 18:45:46.137335 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.137315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"]
Sep 16 18:45:46.138100 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.138017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:45:46.146669 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.146141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"]
Sep 16 18:45:46.146669 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.146176    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9sbcb"]
Sep 16 18:45:46.148100 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.148081    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8bh4h"]
Sep 16 18:45:46.148206 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.148191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.150250 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.150233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.151895 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.151861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z45rg\""
Sep 16 18:45:46.151991 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.151862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:46.151991 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.151916    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:46.151991 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.151965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:46.152772 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.152759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:46.152938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.152922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:46.152996 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.152956    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\""
Sep 16 18:45:46.158776 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.158748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sbcb"]
Sep 16 18:45:46.163857 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.163154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8bh4h"]
Sep 16 18:45:46.283296 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-tmp-dir\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.283296 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-tmp\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.283296 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqjw\" (UniqueName: \"kubernetes.io/projected/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-kube-api-access-zwqjw\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.283495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.283495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.283495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.283495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.283495 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4a58c97-7aae-48ff-b8c5-10a7948a853f-cert\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.283725 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbrt\" (UniqueName: \"kubernetes.io/projected/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-kube-api-access-xrbrt\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.283725 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-config-volume\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.283725 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.283725 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vsz\" (UniqueName: \"kubernetes.io/projected/e4a58c97-7aae-48ff-b8c5-10a7948a853f-kube-api-access-t4vsz\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9j2\" (UniqueName: \"kubernetes.io/projected/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-kube-api-access-md9j2\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.283930 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5brc\" (UniqueName: \"kubernetes.io/projected/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-kube-api-access-v5brc\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.283990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.284006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.284020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrzp\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.284044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.284064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.284158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.284086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.385043 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.385190 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.385190 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.385309 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4a58c97-7aae-48ff-b8c5-10a7948a853f-cert\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.385309 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbrt\" (UniqueName: \"kubernetes.io/projected/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-kube-api-access-xrbrt\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.385309 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-config-volume\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.385441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.385441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vsz\" (UniqueName: \"kubernetes.io/projected/e4a58c97-7aae-48ff-b8c5-10a7948a853f-kube-api-access-t4vsz\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.385441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.385441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.385441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9j2\" (UniqueName: \"kubernetes.io/projected/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-kube-api-access-md9j2\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5brc\" (UniqueName: \"kubernetes.io/projected/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-kube-api-access-v5brc\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.385644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.385573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.385963 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.385940    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:46.386027 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.385966    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:45:46.386078 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.386039    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.886019272 +0000 UTC m=+35.624594750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:45:46.386282 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.386389 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrzp\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.386389 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.386389 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.386540 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.386540 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-tmp-dir\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.386540 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.386803 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-tmp\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.386803 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-config-volume\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.386956 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.386887    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:46.386956 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.386937    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.88692108 +0000 UTC m=+35.625496573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:46.387219 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.387195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.387540 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.387517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-tmp-dir\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.388210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.386483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-tmp\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.388210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.387678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqjw\" (UniqueName: \"kubernetes.io/projected/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-kube-api-access-zwqjw\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.388210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.387720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.390021 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.389998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.390605 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.390549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.390771 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.390746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.390869 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.390759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4a58c97-7aae-48ff-b8c5-10a7948a853f-cert\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.390869 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.390846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.391187 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.391166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-ca\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.391386 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.391316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.391878 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.391852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.392205 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.392168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.392874 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.392849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.395960 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.395919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbrt\" (UniqueName: \"kubernetes.io/projected/06f87f6b-ae2f-41e6-9baf-3a21ae113ccd-kube-api-access-xrbrt\") pod \"managed-serviceaccount-addon-agent-7c778d45f6-xv8mn\" (UID: \"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.397351 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.397326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9j2\" (UniqueName: \"kubernetes.io/projected/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-kube-api-access-md9j2\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.397955 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.397934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vsz\" (UniqueName: \"kubernetes.io/projected/e4a58c97-7aae-48ff-b8c5-10a7948a853f-kube-api-access-t4vsz\") pod \"ingress-canary-9sbcb\" (UID: \"e4a58c97-7aae-48ff-b8c5-10a7948a853f\") " pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.398042 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.398016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5brc\" (UniqueName: \"kubernetes.io/projected/b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153-kube-api-access-v5brc\") pod \"klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt\" (UID: \"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.398587 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.398545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqjw\" (UniqueName: \"kubernetes.io/projected/0f0ed547-c5bd-4a81-a5bf-91dcf8df131c-kube-api-access-zwqjw\") pod \"cluster-proxy-proxy-agent-6d77d4d679-vzfd2\" (UID: \"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.399900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.399878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.400045 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.400024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrzp\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.439890 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.439866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:46.456933 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.456913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:45:46.474566 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.474549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:45:46.480258 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.480239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sbcb"
Sep 16 18:45:46.894120 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.893959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:46.894228 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:46.894136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:46.894290 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.894101    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:46.894290 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.894245    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:45:46.894290 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.894256    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:46.894431 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.894307    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.894289786 +0000 UTC m=+36.632865262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:45:46.894605 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:46.894587    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.894315241 +0000 UTC m=+36.632890717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:47.059707 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.059684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sbcb"]
Sep 16 18:45:47.060392 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.060369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"]
Sep 16 18:45:47.070904 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.070882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"]
Sep 16 18:45:47.074795 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.074775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"]
Sep 16 18:45:47.078892 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:47.078870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0ed547_c5bd_4a81_a5bf_91dcf8df131c.slice/crio-ba8afc5c9027767175d96dfcad65ab27887a7cf024506f7a2550c7299216fff0 WatchSource:0}: Error finding container ba8afc5c9027767175d96dfcad65ab27887a7cf024506f7a2550c7299216fff0: Status 404 returned error can't find the container with id ba8afc5c9027767175d96dfcad65ab27887a7cf024506f7a2550c7299216fff0
Sep 16 18:45:47.079547 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:47.079492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c0ef2a_5cf9_4a6d_93c5_4e4ef1975153.slice/crio-157f9b89bfaf6e0b300c22e64b0fac927578c0bc31ececfbd8a35f4b2cd8c27d WatchSource:0}: Error finding container 157f9b89bfaf6e0b300c22e64b0fac927578c0bc31ececfbd8a35f4b2cd8c27d: Status 404 returned error can't find the container with id 157f9b89bfaf6e0b300c22e64b0fac927578c0bc31ececfbd8a35f4b2cd8c27d
Sep 16 18:45:47.080595 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:47.080452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f87f6b_ae2f_41e6_9baf_3a21ae113ccd.slice/crio-0122a42a542c4ff677e757657ee76c6d2fd70900eca18746766cdc56fd658b6a WatchSource:0}: Error finding container 0122a42a542c4ff677e757657ee76c6d2fd70900eca18746766cdc56fd658b6a: Status 404 returned error can't find the container with id 0122a42a542c4ff677e757657ee76c6d2fd70900eca18746766cdc56fd658b6a
Sep 16 18:45:47.081415 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:45:47.081338    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a58c97_7aae_48ff_b8c5_10a7948a853f.slice/crio-4d61f6f019ed24be9f337a4ccb486311a5fadc2f3c462ade1be90b67fd45532b WatchSource:0}: Error finding container 4d61f6f019ed24be9f337a4ccb486311a5fadc2f3c462ade1be90b67fd45532b: Status 404 returned error can't find the container with id 4d61f6f019ed24be9f337a4ccb486311a5fadc2f3c462ade1be90b67fd45532b
Sep 16 18:45:47.718977 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.718935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:45:47.719910 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.718947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:45:47.719910 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.719114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:45:47.724546 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.724486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\""
Sep 16 18:45:47.732918 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.732900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:47.733241 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.733207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:47.733401 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.733385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\""
Sep 16 18:45:47.733590 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.733574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:47.733911 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.733895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:47.900788 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.900625    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="9b59a91ce31fe585a1873ca8e6c41a5a81646099f1e665d327b9e94e0f7e8359" exitCode=0
Sep 16 18:45:47.900788 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.900725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"9b59a91ce31fe585a1873ca8e6c41a5a81646099f1e665d327b9e94e0f7e8359"}
Sep 16 18:45:47.902132 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.902108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:47.902326 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.902154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:47.902326 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:47.902293    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:47.902326 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:47.902309    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:45:47.902710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:47.902540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.902515207 +0000 UTC m=+38.641090695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:45:47.902710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:47.902591    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:47.902710 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:47.902644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.902627625 +0000 UTC m=+38.641203104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:47.904180 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.903953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" event={"ID":"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153","Type":"ContainerStarted","Data":"157f9b89bfaf6e0b300c22e64b0fac927578c0bc31ececfbd8a35f4b2cd8c27d"}
Sep 16 18:45:47.906342 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.906280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerStarted","Data":"ba8afc5c9027767175d96dfcad65ab27887a7cf024506f7a2550c7299216fff0"}
Sep 16 18:45:47.910520 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.908454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sbcb" event={"ID":"e4a58c97-7aae-48ff-b8c5-10a7948a853f","Type":"ContainerStarted","Data":"4d61f6f019ed24be9f337a4ccb486311a5fadc2f3c462ade1be90b67fd45532b"}
Sep 16 18:45:47.911710 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:47.911682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn" event={"ID":"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd","Type":"ContainerStarted","Data":"0122a42a542c4ff677e757657ee76c6d2fd70900eca18746766cdc56fd658b6a"}
Sep 16 18:45:48.928237 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:48.927442    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9c30-89bc-4ca4-8202-f77f3871e8ae" containerID="065296e8cdc409c24c3bed16ca19398827b95f147441cecc0235ed1ff2a3d32a" exitCode=0
Sep 16 18:45:48.928237 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:48.927509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerDied","Data":"065296e8cdc409c24c3bed16ca19398827b95f147441cecc0235ed1ff2a3d32a"}
Sep 16 18:45:49.921755 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:49.921669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:49.921755 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:49.921731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:49.921988 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:49.921860    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:49.921988 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:49.921883    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:45:49.921988 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:49.921907    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:49.921988 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:49.921950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:53.921930292 +0000 UTC m=+42.660505789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:45:49.921988 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:49.921968    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:53.92195942 +0000 UTC m=+42.660534898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:49.937107 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:49.937080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vznp7" event={"ID":"2c6d9c30-89bc-4ca4-8202-f77f3871e8ae","Type":"ContainerStarted","Data":"b8614a722c7c3a02bddff56087d736703d64d4fb614c3e01620d699d56590901"}
Sep 16 18:45:49.967944 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:49.967889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vznp7" podStartSLOduration=8.87554854 podStartE2EDuration="37.96786779s" totalImagesPullingTime="29.09231925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.966722696 +0000 UTC m=+38.705298194" watchObservedRunningTime="2026-09-16 18:45:49.96786779 +0000 UTC m=+38.706443289"
Sep 16 18:45:50.154536 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:50.154498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.304768 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:50.304687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.563543 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:50.563465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.701766 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:50.701738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:51.425666 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:51.425639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:51.505825 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:51.505789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:52.763680 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:52.763641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:52.871717 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:52.871685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:53.954967 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:53.954930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:53.954983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:53.955112    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:53.955133    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:53.955142    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:53.955203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:01.955181124 +0000 UTC m=+50.693756602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:45:53.955396 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:45:53.955226    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:01.955215762 +0000 UTC m=+50.693791239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:55.954451 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.954386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sbcb" event={"ID":"e4a58c97-7aae-48ff-b8c5-10a7948a853f","Type":"ContainerStarted","Data":"b65fea6d994575cf5c887ee9dd7be1acdd1e4c8065773b6a3b745096001a4208"}
Sep 16 18:45:55.955983 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.955952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn" event={"ID":"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd","Type":"ContainerStarted","Data":"74d5cefed6d02489688015795e0296905e8eb3e35570153c68863c2d44dd2234"}
Sep 16 18:45:55.957367 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.957343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" event={"ID":"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153","Type":"ContainerStarted","Data":"bda7b30d3f3cc330d4d23fe632b0f52c888458d8f8c70635601f4737b45e1ddc"}
Sep 16 18:45:55.957513 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.957494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:55.959346 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.959327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:45:55.959682 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.959659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerStarted","Data":"c28e8f7b9c4262f04a49039db8e1b4c55ea4090c9136e71a9fc02e9a1b680e61"}
Sep 16 18:45:55.959748 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.959696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerStarted","Data":"6c361ae3f47e77a598c620d9e51693802d7e670ac50886966e0a86bd14cedf13"}
Sep 16 18:45:55.959748 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:55.959716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerStarted","Data":"817a25a4e4420ebaca5ba4b79ebdfc4f1f1d1c9621a9a079505eed2ae3bd317c"}
Sep 16 18:45:56.007798 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:56.007760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9sbcb" podStartSLOduration=1.780694659 podStartE2EDuration="10.00774701s" totalImagesPullingTime="8.227052351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:55.983940591 +0000 UTC m=+44.722516090" watchObservedRunningTime="2026-09-16 18:45:56.00774701 +0000 UTC m=+44.746322542"
Sep 16 18:45:56.035507 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:56.035471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" podStartSLOduration=25.802355332 podStartE2EDuration="34.035460447s" totalImagesPullingTime="8.233105115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.035013171 +0000 UTC m=+44.773588670" watchObservedRunningTime="2026-09-16 18:45:56.035460447 +0000 UTC m=+44.774035961"
Sep 16 18:45:56.035598 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:56.035539    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn" podStartSLOduration=25.808467844 podStartE2EDuration="34.035533183s" totalImagesPullingTime="8.227065339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.007745995 +0000 UTC m=+44.746321493" watchObservedRunningTime="2026-09-16 18:45:56.035533183 +0000 UTC m=+44.774108680"
Sep 16 18:45:56.064090 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:56.064055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" podStartSLOduration=25.817675479000002 podStartE2EDuration="34.06404408s" totalImagesPullingTime="8.246368601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.061563146 +0000 UTC m=+44.800138645" watchObservedRunningTime="2026-09-16 18:45:56.06404408 +0000 UTC m=+44.802619578"
Sep 16 18:45:58.395626 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:58.395596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:58.458009 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:45:58.457985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.010419 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:02.010378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:46:02.010419 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:02.010422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:46:02.011028 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:02.010517    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:02.011028 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:02.010537    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:46:02.011028 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:02.010537    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.011028 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:02.010608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.01058969 +0000 UTC m=+66.749165177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.011028 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:02.010628    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.010619353 +0000 UTC m=+66.749194828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:46:02.608780 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:02.608749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.706406 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:02.706374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.613158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:03.613121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.682106 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:03.682079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:10.056983 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:10.056949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:11.881377 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:11.881353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xdr6v"
Sep 16 18:46:12.052752 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:12.052699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.126856 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:12.126827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.437088 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:12.437064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.510131 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:12.510108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.519243 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.519205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:46:17.519684 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.519267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:46:17.524269 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.524233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:17.524757 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.524438    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:17.530104 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:17.530077    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:17.530222 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:17.530195    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:21.530177374 +0000 UTC m=+130.268752887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : secret "metrics-daemon-secret" not found
Sep 16 18:46:17.533709 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.533678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3aa8d45-86ee-44c6-b9dc-01e878c909b5-original-pull-secret\") pod \"global-pull-secret-syncer-57jlt\" (UID: \"c3aa8d45-86ee-44c6-b9dc-01e878c909b5\") " pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:46:17.620004 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.619971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:46:17.623496 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.623475    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:17.633911 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.633894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:17.643171 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.643136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5pr\" (UniqueName: \"kubernetes.io/projected/f9afdf29-14a2-48be-8732-7981a9c78d80-kube-api-access-mv5pr\") pod \"network-check-target-jmbzj\" (UID: \"f9afdf29-14a2-48be-8732-7981a9c78d80\") " pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:46:17.725763 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.725739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.750126 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.750104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57jlt"
Sep 16 18:46:17.761326 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.761300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\""
Sep 16 18:46:17.769076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.769056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:46:17.817125 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:17.817090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.022005 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:18.021977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:46:18.022116 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:18.022014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:46:18.022158 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:18.022126    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.022203 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:18.022193    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:50.022179461 +0000 UTC m=+98.760754936 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.022203 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:18.022129    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:18.022273 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:18.022212    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:46:18.022273 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:18.022242    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:50.022232367 +0000 UTC m=+98.760807846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:46:18.061959 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:18.061935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57jlt"]
Sep 16 18:46:18.063931 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:46:18.063909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3aa8d45_86ee_44c6_b9dc_01e878c909b5.slice/crio-1cbe49e5f3df4d8302d3553c5eb62f34e6a18ad0199db8a4f432bf132f9cb80a WatchSource:0}: Error finding container 1cbe49e5f3df4d8302d3553c5eb62f34e6a18ad0199db8a4f432bf132f9cb80a: Status 404 returned error can't find the container with id 1cbe49e5f3df4d8302d3553c5eb62f34e6a18ad0199db8a4f432bf132f9cb80a
Sep 16 18:46:18.087382 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:18.087361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmbzj"]
Sep 16 18:46:18.089737 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:46:18.089705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9afdf29_14a2_48be_8732_7981a9c78d80.slice/crio-2ad1aef3f1fcb44aae913bf5ebd39d00dc624a9e14a7fcfa27e864ad6b8bec76 WatchSource:0}: Error finding container 2ad1aef3f1fcb44aae913bf5ebd39d00dc624a9e14a7fcfa27e864ad6b8bec76: Status 404 returned error can't find the container with id 2ad1aef3f1fcb44aae913bf5ebd39d00dc624a9e14a7fcfa27e864ad6b8bec76
Sep 16 18:46:19.031762 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:19.031721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57jlt" event={"ID":"c3aa8d45-86ee-44c6-b9dc-01e878c909b5","Type":"ContainerStarted","Data":"1cbe49e5f3df4d8302d3553c5eb62f34e6a18ad0199db8a4f432bf132f9cb80a"}
Sep 16 18:46:19.033420 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:19.033390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmbzj" event={"ID":"f9afdf29-14a2-48be-8732-7981a9c78d80","Type":"ContainerStarted","Data":"2ad1aef3f1fcb44aae913bf5ebd39d00dc624a9e14a7fcfa27e864ad6b8bec76"}
Sep 16 18:46:19.639917 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:19.639880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:19.712614 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:19.712586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.835088 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:20.835052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.936170 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:20.936137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.048712 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:23.048671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmbzj" event={"ID":"f9afdf29-14a2-48be-8732-7981a9c78d80","Type":"ContainerStarted","Data":"2a71962fd7e626551c26b8d291844aa88564662d0acc823b4841d5c4dea27cb3"}
Sep 16 18:46:23.049225 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:23.048760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:46:23.050560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:23.050525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57jlt" event={"ID":"c3aa8d45-86ee-44c6-b9dc-01e878c909b5","Type":"ContainerStarted","Data":"31bb7f07fe9ade63bcbfe2eca6fc4d1c06fa81d0e6636b59d7db77ce3d7c2831"}
Sep 16 18:46:23.069189 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:23.069132    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jmbzj" podStartSLOduration=66.719073986 podStartE2EDuration="1m11.069115378s" totalImagesPullingTime="4.350041392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:23.067767174 +0000 UTC m=+71.806342671" watchObservedRunningTime="2026-09-16 18:46:23.069115378 +0000 UTC m=+71.807690875"
Sep 16 18:46:23.086615 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:23.086568    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-57jlt" podStartSLOduration=66.707342489 podStartE2EDuration="1m11.086553176s" totalImagesPullingTime="4.379210687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:23.086064906 +0000 UTC m=+71.824640406" watchObservedRunningTime="2026-09-16 18:46:23.086553176 +0000 UTC m=+71.825128674"
Sep 16 18:46:40.637509 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:40.637368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.883237 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:41.883209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.037581 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:50.037535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:50.037598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:50.037697    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:50.037714    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb884bd5d-86q7l: secret "image-registry-tls" not found
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:50.037777    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:50.037799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls podName:87221153-8517-4ad0-9df3-c252e1868753 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:54.037784027 +0000 UTC m=+162.776359507 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls") pod "image-registry-7cb884bd5d-86q7l" (UID: "87221153-8517-4ad0-9df3-c252e1868753") : secret "image-registry-tls" not found
Sep 16 18:46:50.038035 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:46:50.037868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls podName:f0b6d71b-55f4-4abf-a549-b2e7a236b7b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:54.037849974 +0000 UTC m=+162.776425505 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls") pod "dns-default-8bh4h" (UID: "f0b6d71b-55f4-4abf-a549-b2e7a236b7b0") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:52.263491 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:52.263451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:52.374679 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:52.374636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.055809 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:46:54.055784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jmbzj"
Sep 16 18:47:21.550726 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:21.550683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:47:21.551245 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:21.550809    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:47:21.551245 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:21.550890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs podName:1dd2fed3-b044-4df8-9a04-0f1de72ac07c nodeName:}" failed. No retries permitted until 2026-09-16 18:49:23.550875735 +0000 UTC m=+252.289451211 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs") pod "network-metrics-daemon-mmtgl" (UID: "1dd2fed3-b044-4df8-9a04-0f1de72ac07c") : secret "metrics-daemon-secret" not found
Sep 16 18:47:33.407793 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:33.407759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:33.517117 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:33.517086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.743138 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:35.743098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.840719 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:35.840687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.151764 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.151729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.319343 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.319311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.424673 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.424597    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6fkgr"]
Sep 16 18:47:37.426860 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.426841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.430232 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.430202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:37.431733 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.431710    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:47:37.431868 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.431733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:47:37.431868 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.431791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fk2dc\""
Sep 16 18:47:37.432149 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.432124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:37.451081 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.451049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.460646 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.460622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fkgr"]
Sep 16 18:47:37.563671 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.563643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.563775 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.563676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqnj\" (UniqueName: \"kubernetes.io/projected/a9550f81-9761-40cd-a78d-68d965355415-kube-api-access-4fqnj\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.563775 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.563707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9550f81-9761-40cd-a78d-68d965355415-crio-socket\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.563870 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.563795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9550f81-9761-40cd-a78d-68d965355415-data-volume\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.563870 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.563844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.629152 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.629123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.664600 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9550f81-9761-40cd-a78d-68d965355415-data-volume\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664841 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:37.664774    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:37.664841 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:37.664782    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:37.664841 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqnj\" (UniqueName: \"kubernetes.io/projected/a9550f81-9761-40cd-a78d-68d965355415-kube-api-access-4fqnj\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664994 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:37.664879    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.164860314 +0000 UTC m=+146.903435821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:37.664994 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9550f81-9761-40cd-a78d-68d965355415-data-volume\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664994 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:37.664908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.164900998 +0000 UTC m=+146.903476478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:37.664994 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9550f81-9761-40cd-a78d-68d965355415-crio-socket\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.664994 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.664983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9550f81-9761-40cd-a78d-68d965355415-crio-socket\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.684032 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.683980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqnj\" (UniqueName: \"kubernetes.io/projected/a9550f81-9761-40cd-a78d-68d965355415-kube-api-access-4fqnj\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:37.723031 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.723006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.865879 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.865847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.970164 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:37.970098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:38.167965 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:38.167935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:38.168362 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:38.167995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:38.168362 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:38.168069    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:38.168362 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:38.168090    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:38.168362 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:38.168147    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.168127992 +0000 UTC m=+147.906703506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:38.168362 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:38.168168    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.168158286 +0000 UTC m=+147.906733762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:39.174250 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.174217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:39.174732 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.174272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:39.174732 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:39.174354    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:39.174732 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:39.174395    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:39.174732 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:39.174404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.174392079 +0000 UTC m=+149.912967554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:39.174732 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:39.174474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.174452692 +0000 UTC m=+149.913028168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:39.526527 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.526435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:39.642563 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.642531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:39.890807 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.890776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:39.964734 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:39.964698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.336568 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:40.336499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s8xph_14610c4a-a5e4-4bbc-a969-333868322d1c/dns-node-resolver/0.log"
Sep 16 18:47:40.548522 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:40.548491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:41.133151 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:41.133121    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mq86t_0672e4fc-19cc-464f-9123-8b90a9ce0107/node-ca/0.log"
Sep 16 18:47:41.187524 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:41.187499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:41.187625 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:41.187554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:41.187676 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:41.187640    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:41.187711 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:41.187699    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.187686605 +0000 UTC m=+153.926262081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:41.187764 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:41.187641    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:41.187800 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:41.187789    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.187775983 +0000 UTC m=+153.926351463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:41.783176 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:41.783143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:42.139342 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:42.139316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9sbcb_e4a58c97-7aae-48ff-b8c5-10a7948a853f/serve-healthcheck-canary/0.log"
Sep 16 18:47:43.168512 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:43.168477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.395426 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:43.395363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.536369 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:43.536288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.634257 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:43.634230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.734137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:43.734111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.010434 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:44.010402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.745347 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:44.745319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.215164 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.215128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:45.215315 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.215191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:45.215315 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:45.215293    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:45.215431 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:45.215308    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:45.215431 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:45.215351    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.215336723 +0000 UTC m=+161.953912199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:45.215431 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:45.215382    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls podName:a9550f81-9761-40cd-a78d-68d965355415 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.215360895 +0000 UTC m=+161.953936406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fkgr" (UID: "a9550f81-9761-40cd-a78d-68d965355415") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:45.314429 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.314402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.513064 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.512999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.695180 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.695150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.904008 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:45.903971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:46.900942 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:46.900912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:46.976570 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:46.976541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.070779 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:47.070752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.305778 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:47.305714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.512918 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:47.512894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.663830 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:47.663801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.853250 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:47.853225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:48.163248 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:48.163213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:48.723646 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:48.723610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.131500 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:49.131469    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" podUID="87221153-8517-4ad0-9df3-c252e1868753"
Sep 16 18:47:49.186056 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:49.186022    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8bh4h" podUID="f0b6d71b-55f4-4abf-a549-b2e7a236b7b0"
Sep 16 18:47:49.249042 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.249008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.308731 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.308704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:47:49.308858 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.308708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:47:49.343078 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.343049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.493870 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.493778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.695201 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.695141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:49.893386 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:49.893360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.338600 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:50.338505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.739275 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:47:50.739234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mmtgl" podUID="1dd2fed3-b044-4df8-9a04-0f1de72ac07c"
Sep 16 18:47:50.903325 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:50.903296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.113641 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:51.113611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.512908 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:51.512788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.761428 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:51.761397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:52.106865 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:52.106833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:52.732442 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:52.732410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.275669 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.275634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:53.275875 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.275705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:53.276252 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.276228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9550f81-9761-40cd-a78d-68d965355415-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:53.277868 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.277851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9550f81-9761-40cd-a78d-68d965355415-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fkgr\" (UID: \"a9550f81-9761-40cd-a78d-68d965355415\") " pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:53.327616 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.327586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.336520 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.336499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fkgr"
Sep 16 18:47:53.510594 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.510566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.617805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.617774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fkgr"]
Sep 16 18:47:53.621649 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:47:53.621625    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9550f81_9761_40cd_a78d_68d965355415.slice/crio-47ba0f6f0c10969f4052d879e4be5f23460b3264d84d9ea75ebc1ad19bab0b0a WatchSource:0}: Error finding container 47ba0f6f0c10969f4052d879e4be5f23460b3264d84d9ea75ebc1ad19bab0b0a: Status 404 returned error can't find the container with id 47ba0f6f0c10969f4052d879e4be5f23460b3264d84d9ea75ebc1ad19bab0b0a
Sep 16 18:47:53.690327 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.690299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.933877 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:53.933776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:54.081697 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.081662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:47:54.081845 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.081733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:47:54.084257 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.084235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0b6d71b-55f4-4abf-a549-b2e7a236b7b0-metrics-tls\") pod \"dns-default-8bh4h\" (UID: \"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0\") " pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:47:54.084435 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.084417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"image-registry-7cb884bd5d-86q7l\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") " pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:47:54.116765 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.116735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\""
Sep 16 18:47:54.116765 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.116735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s9n6t\""
Sep 16 18:47:54.119841 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.119805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:47:54.119926 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.119854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:47:54.325608 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.325577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fkgr" event={"ID":"a9550f81-9761-40cd-a78d-68d965355415","Type":"ContainerStarted","Data":"35bcfae949d007819c7b11f969803d983cacaefb248b90ac75080757ff1e455b"}
Sep 16 18:47:54.325706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.325631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fkgr" event={"ID":"a9550f81-9761-40cd-a78d-68d965355415","Type":"ContainerStarted","Data":"622cc28a3982642871b9bfa8c7e27f36fe68eb558c0f76b0f8a26eda5a988168"}
Sep 16 18:47:54.325706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.325650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fkgr" event={"ID":"a9550f81-9761-40cd-a78d-68d965355415","Type":"ContainerStarted","Data":"47ba0f6f0c10969f4052d879e4be5f23460b3264d84d9ea75ebc1ad19bab0b0a"}
Sep 16 18:47:54.448692 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.448669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8bh4h"]
Sep 16 18:47:54.450475 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:47:54.450449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b6d71b_55f4_4abf_a549_b2e7a236b7b0.slice/crio-9ab98c1a5f7d83a30d36a58244a01fe07015fd717f2a825a2d0569830074155b WatchSource:0}: Error finding container 9ab98c1a5f7d83a30d36a58244a01fe07015fd717f2a825a2d0569830074155b: Status 404 returned error can't find the container with id 9ab98c1a5f7d83a30d36a58244a01fe07015fd717f2a825a2d0569830074155b
Sep 16 18:47:54.453499 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:54.453476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:47:54.455891 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:47:54.455871    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87221153_8517_4ad0_9df3_c252e1868753.slice/crio-4d852a6adea598f4874b11ebce3df6753ab9867ed2846bb2d070946de5a3b07f WatchSource:0}: Error finding container 4d852a6adea598f4874b11ebce3df6753ab9867ed2846bb2d070946de5a3b07f: Status 404 returned error can't find the container with id 4d852a6adea598f4874b11ebce3df6753ab9867ed2846bb2d070946de5a3b07f
Sep 16 18:47:55.332101 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.332047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bh4h" event={"ID":"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0","Type":"ContainerStarted","Data":"9ab98c1a5f7d83a30d36a58244a01fe07015fd717f2a825a2d0569830074155b"}
Sep 16 18:47:55.333857 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.333829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" event={"ID":"87221153-8517-4ad0-9df3-c252e1868753","Type":"ContainerStarted","Data":"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"}
Sep 16 18:47:55.333981 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.333875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" event={"ID":"87221153-8517-4ad0-9df3-c252e1868753","Type":"ContainerStarted","Data":"4d852a6adea598f4874b11ebce3df6753ab9867ed2846bb2d070946de5a3b07f"}
Sep 16 18:47:55.334052 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.333976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:47:55.377460 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.377395    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" podStartSLOduration=162.377376359 podStartE2EDuration="2m42.377376359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.377156515 +0000 UTC m=+164.115732014" watchObservedRunningTime="2026-09-16 18:47:55.377376359 +0000 UTC m=+164.115951861"
Sep 16 18:47:55.958546 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:55.958497    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" podUID="b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused"
Sep 16 18:47:56.338345 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.338315    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153" containerID="bda7b30d3f3cc330d4d23fe632b0f52c888458d8f8c70635601f4737b45e1ddc" exitCode=1
Sep 16 18:47:56.338713 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.338392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" event={"ID":"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153","Type":"ContainerDied","Data":"bda7b30d3f3cc330d4d23fe632b0f52c888458d8f8c70635601f4737b45e1ddc"}
Sep 16 18:47:56.338792 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.338755    2563 scope.go:117] "RemoveContainer" containerID="bda7b30d3f3cc330d4d23fe632b0f52c888458d8f8c70635601f4737b45e1ddc"
Sep 16 18:47:56.340284 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.340260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bh4h" event={"ID":"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0","Type":"ContainerStarted","Data":"ce7816c48623a93f962352a4b7f9ec3f48c84cb4a7da04ef3e087b134138bde4"}
Sep 16 18:47:56.340362 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.340335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:47:56.340418 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.340364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8bh4h" event={"ID":"f0b6d71b-55f4-4abf-a549-b2e7a236b7b0","Type":"ContainerStarted","Data":"6e9f8dc2ff5e2fcac3e79ae0ae8a36d61e29cc1377f604781ec30237b951b24e"}
Sep 16 18:47:56.341709 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.341691    2563 generic.go:356] "Generic (PLEG): container finished" podID="06f87f6b-ae2f-41e6-9baf-3a21ae113ccd" containerID="74d5cefed6d02489688015795e0296905e8eb3e35570153c68863c2d44dd2234" exitCode=255
Sep 16 18:47:56.341784 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.341761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn" event={"ID":"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd","Type":"ContainerDied","Data":"74d5cefed6d02489688015795e0296905e8eb3e35570153c68863c2d44dd2234"}
Sep 16 18:47:56.342048 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.342029    2563 scope.go:117] "RemoveContainer" containerID="74d5cefed6d02489688015795e0296905e8eb3e35570153c68863c2d44dd2234"
Sep 16 18:47:56.343765 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.343744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fkgr" event={"ID":"a9550f81-9761-40cd-a78d-68d965355415","Type":"ContainerStarted","Data":"da7dc6b7cf182533ee9ee689cb05a8cad99fd9bace7468481b11d4cf4a5e8d04"}
Sep 16 18:47:56.440503 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.440445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:47:56.450418 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.450368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6fkgr" podStartSLOduration=17.13813593 podStartE2EDuration="19.450348994s" totalImagesPullingTime="2.312213064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.417645968 +0000 UTC m=+165.156221468" watchObservedRunningTime="2026-09-16 18:47:56.450348994 +0000 UTC m=+165.188924493"
Sep 16 18:47:56.451142 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.451114    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8bh4h" podStartSLOduration=128.842710826 podStartE2EDuration="2m10.451104224s" totalImagesPullingTime="1.608393398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.450284946 +0000 UTC m=+165.188860443" watchObservedRunningTime="2026-09-16 18:47:56.451104224 +0000 UTC m=+165.189679721"
Sep 16 18:47:56.457076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:56.457057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn"
Sep 16 18:47:57.349669 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:57.349627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt" event={"ID":"b8c0ef2a-5cf9-4a6d-93c5-4e4ef1975153","Type":"ContainerStarted","Data":"1a52c14703eab909f8ce91eaa920d615d4364263dbc998441fd9ae4683bcfbed"}
Sep 16 18:47:57.350217 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:57.349799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:47:57.350950 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:57.350928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c6cc8bbb6-cdxjt"
Sep 16 18:47:57.351879 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:57.351851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c778d45f6-xv8mn" event={"ID":"06f87f6b-ae2f-41e6-9baf-3a21ae113ccd","Type":"ContainerStarted","Data":"1891d15e47a4bda58ee7d9b4d00768592dc336903fb2fa7a84c7c9a4bd60b387"}
Sep 16 18:47:58.403649 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:58.403620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:58.478158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:58.478132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:59.967805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:47:59.967773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:00.049998 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:00.049969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:06.354231 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:06.354155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8bh4h"
Sep 16 18:48:06.718314 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:06.718275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:48:07.615750 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:07.615717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:07.726782 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:07.726748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:07.894235 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:07.894163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:08.068578 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:08.068549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:08.170534 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:08.170465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:08.239226 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:08.239197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:09.941922 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:09.941887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:10.016798 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:10.016771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:10.423703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:10.423672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:12.771644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:12.771607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.124256 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.124218    2563 patch_prober.go:28] interesting pod/image-registry-7cb884bd5d-86q7l container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:48:14.124624 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.124280    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" podUID="87221153-8517-4ad0-9df3-c252e1868753" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:14.135719 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.135692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.262925 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.262897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.408643 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.408579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.434716 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.434692    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d7gkw"]
Sep 16 18:48:14.438062 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.438045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.441547 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.441528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:48:14.442385 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.442368    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:48:14.443850 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.443832    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:48:14.444248 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.444232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrnx\""
Sep 16 18:48:14.444336 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.444236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:48:14.444336 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.444259    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:48:14.444789 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.444592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:48:14.518439 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.518395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.521594 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-metrics-client-ca\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-textfile\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-root\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-sys\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-wtmp\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrxz\" (UniqueName: \"kubernetes.io/projected/f1667821-9b9d-4315-b29f-cda37b4c7fe2-kube-api-access-jwrxz\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.521900 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.521893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622641 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-textfile\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622722 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-root\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622722 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-sys\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622722 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-wtmp\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622722 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-sys\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-root\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrxz\" (UniqueName: \"kubernetes.io/projected/f1667821-9b9d-4315-b29f-cda37b4c7fe2-kube-api-access-jwrxz\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:48:14.622852    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-wtmp\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.622938 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:48:14.622917    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls podName:f1667821-9b9d-4315-b29f-cda37b4c7fe2 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:15.122897242 +0000 UTC m=+183.861472736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls") pod "node-exporter-d7gkw" (UID: "f1667821-9b9d-4315-b29f-cda37b4c7fe2") : secret "node-exporter-tls" not found
Sep 16 18:48:14.623225 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-metrics-client-ca\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.623225 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.622997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-textfile\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.623356 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.623338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.623390 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.623347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1667821-9b9d-4315-b29f-cda37b4c7fe2-metrics-client-ca\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.625137 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.625121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.634828 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.634793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrxz\" (UniqueName: \"kubernetes.io/projected/f1667821-9b9d-4315-b29f-cda37b4c7fe2-kube-api-access-jwrxz\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:14.688840 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.688762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:14.787674 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:14.787643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:15.052259 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.052191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:15.127333 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.127305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:15.129478 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.129458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1667821-9b9d-4315-b29f-cda37b4c7fe2-node-exporter-tls\") pod \"node-exporter-d7gkw\" (UID: \"f1667821-9b9d-4315-b29f-cda37b4c7fe2\") " pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:15.228533 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.228508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:15.347428 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.347379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d7gkw"
Sep 16 18:48:15.358385 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:48:15.358362    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1667821_9b9d_4315_b29f_cda37b4c7fe2.slice/crio-e1bdaf08d3b91f144acfe9958da99a1390be638b48702e9395d28da01cecf33f WatchSource:0}: Error finding container e1bdaf08d3b91f144acfe9958da99a1390be638b48702e9395d28da01cecf33f: Status 404 returned error can't find the container with id e1bdaf08d3b91f144acfe9958da99a1390be638b48702e9395d28da01cecf33f
Sep 16 18:48:15.414351 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.414329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7gkw" event={"ID":"f1667821-9b9d-4315-b29f-cda37b4c7fe2","Type":"ContainerStarted","Data":"e1bdaf08d3b91f144acfe9958da99a1390be638b48702e9395d28da01cecf33f"}
Sep 16 18:48:15.770320 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:15.770290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:16.001951 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:16.001915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:16.348033 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:16.348014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:48:16.419757 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:16.419726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7gkw" event={"ID":"f1667821-9b9d-4315-b29f-cda37b4c7fe2","Type":"ContainerStarted","Data":"c97121c0e25961724b9a323d96652083d59fc0c85bf3face12f1c63ec08edc0f"}
Sep 16 18:48:17.159268 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.159232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:17.316655 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.316619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:17.425095 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.425017    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1667821-9b9d-4315-b29f-cda37b4c7fe2" containerID="c97121c0e25961724b9a323d96652083d59fc0c85bf3face12f1c63ec08edc0f" exitCode=0
Sep 16 18:48:17.425470 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.425093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7gkw" event={"ID":"f1667821-9b9d-4315-b29f-cda37b4c7fe2","Type":"ContainerDied","Data":"c97121c0e25961724b9a323d96652083d59fc0c85bf3face12f1c63ec08edc0f"}
Sep 16 18:48:17.522639 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.522613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:17.628241 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.628216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:17.985654 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:17.985627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.147171 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:18.147135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.430125 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:18.430099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7gkw" event={"ID":"f1667821-9b9d-4315-b29f-cda37b4c7fe2","Type":"ContainerStarted","Data":"bdd71c4f42e33b55048f673fc7600d15cc8302d4496eb37347fa2b1d9cd9b877"}
Sep 16 18:48:18.430450 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:18.430152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d7gkw" event={"ID":"f1667821-9b9d-4315-b29f-cda37b4c7fe2","Type":"ContainerStarted","Data":"3a66b4f921ac78320379aa06fae8ce1f81f6ddb6ea9376944615144f3b1eb6f8"}
Sep 16 18:48:18.469516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:18.469468    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d7gkw" podStartSLOduration=3.469959528 podStartE2EDuration="4.469456265s" totalImagesPullingTime="937.847625ms" totalInitContainerRuntime="61.649112ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:18.467502939 +0000 UTC m=+187.206078438" watchObservedRunningTime="2026-09-16 18:48:18.469456265 +0000 UTC m=+187.208031762"
Sep 16 18:48:19.170800 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:19.170769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:19.574616 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:19.574537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:20.908168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:20.908127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:21.740764 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:21.740730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:48:21.854171 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:21.854140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:21.962646 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:21.962622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.643169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:22.643133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.837609 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:22.837577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.942138 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:22.942075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.074664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:23.074636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.305070 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:23.305014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.432885 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:23.432858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.198607 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.198567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.302642 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.302608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.424664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.424635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.525258 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.525186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.632358 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.632333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.721737 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:28.721714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.024613 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:29.024582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.127124 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:29.127093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.228126 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:29.228100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.302336 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:29.302269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.821210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:29.821174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:30.017728 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:30.017702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:30.143260 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:30.143231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:30.853610 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:30.853572    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:36.170443 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:36.170403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:36.271585 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:36.271530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.085025 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:40.084996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.192533 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:40.192505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.051496 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:43.051450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.129012 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:43.128982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.386281 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:43.386249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.457721 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:43.457696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:44.133764 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:44.133729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:44.217843 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:44.217785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:46.476367 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:46.476331    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" podUID="0f0ed547-c5bd-4a81-a5bf-91dcf8df131c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:48:46.759748 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:46.759662    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" podUID="87221153-8517-4ad0-9df3-c252e1868753" containerName="registry" containerID="cri-o://4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420" gracePeriod=30
Sep 16 18:48:46.994728 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:46.994700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:48:47.143353 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143466 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143668 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143668 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143668 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbrzp\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp\") pod \"87221153-8517-4ad0-9df3-c252e1868753\" (UID: \"87221153-8517-4ad0-9df3-c252e1868753\") "
Sep 16 18:48:47.143973 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:47.143973 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.143925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:47.146247 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.146217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:47.146247 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.146228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp" (OuterVolumeSpecName: "kube-api-access-tbrzp") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "kube-api-access-tbrzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:47.146521 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.146491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:47.146584 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.146563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:47.147338 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.147315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:47.155051 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.155025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "87221153-8517-4ad0-9df3-c252e1868753" (UID: "87221153-8517-4ad0-9df3-c252e1868753"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:47.244445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244409    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-registry-certificates\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244445 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244445    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87221153-8517-4ad0-9df3-c252e1868753-trusted-ca\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244454    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-installation-pull-secrets\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244465    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-bound-sa-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244474    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrzp\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-kube-api-access-tbrzp\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244482    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87221153-8517-4ad0-9df3-c252e1868753-image-registry-private-configuration\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244491    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87221153-8517-4ad0-9df3-c252e1868753-registry-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.244560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.244499    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87221153-8517-4ad0-9df3-c252e1868753-ca-trust-extracted\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:48:47.465251 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.465172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.523943 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.523913    2563 generic.go:356] "Generic (PLEG): container finished" podID="87221153-8517-4ad0-9df3-c252e1868753" containerID="4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420" exitCode=0
Sep 16 18:48:47.524299 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.523975    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l"
Sep 16 18:48:47.524299 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.523995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" event={"ID":"87221153-8517-4ad0-9df3-c252e1868753","Type":"ContainerDied","Data":"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"}
Sep 16 18:48:47.524299 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.524051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb884bd5d-86q7l" event={"ID":"87221153-8517-4ad0-9df3-c252e1868753","Type":"ContainerDied","Data":"4d852a6adea598f4874b11ebce3df6753ab9867ed2846bb2d070946de5a3b07f"}
Sep 16 18:48:47.524299 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.524096    2563 scope.go:117] "RemoveContainer" containerID="4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"
Sep 16 18:48:47.533121 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.533101    2563 scope.go:117] "RemoveContainer" containerID="4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"
Sep 16 18:48:47.533433 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:48:47.533413    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420\": container with ID starting with 4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420 not found: ID does not exist" containerID="4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"
Sep 16 18:48:47.533489 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.533441    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420"} err="failed to get container status \"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420\": rpc error: code = NotFound desc = could not find container \"4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420\": container with ID starting with 4ae2ded947e1ac9dbeb99215ebfee7cf90cce0269ee15c02658d3460a2ca6420 not found: ID does not exist"
Sep 16 18:48:47.549316 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.549287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:48:47.555490 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.555464    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cb884bd5d-86q7l"]
Sep 16 18:48:47.555771 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.555742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.722706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:47.722642    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87221153-8517-4ad0-9df3-c252e1868753" path="/var/lib/kubelet/pods/87221153-8517-4ad0-9df3-c252e1868753/volumes"
Sep 16 18:48:50.235935 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:50.235905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/init-textfile/0.log"
Sep 16 18:48:50.436409 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:50.436383    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/node-exporter/0.log"
Sep 16 18:48:50.635850 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:50.635802    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/kube-rbac-proxy/0.log"
Sep 16 18:48:56.476370 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:56.476329    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" podUID="0f0ed547-c5bd-4a81-a5bf-91dcf8df131c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:48:58.418753 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:58.418716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:58.498857 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:48:58.498808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:01.685484 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:01.685440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:01.750135 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:01.750073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:06.475793 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:06.475750    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" podUID="0f0ed547-c5bd-4a81-a5bf-91dcf8df131c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:49:06.476300 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:06.475830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2"
Sep 16 18:49:06.476300 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:06.476240    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c28e8f7b9c4262f04a49039db8e1b4c55ea4090c9136e71a9fc02e9a1b680e61"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:49:06.476300 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:06.476273    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" podUID="0f0ed547-c5bd-4a81-a5bf-91dcf8df131c" containerName="service-proxy" containerID="cri-o://c28e8f7b9c4262f04a49039db8e1b4c55ea4090c9136e71a9fc02e9a1b680e61" gracePeriod=30
Sep 16 18:49:07.594716 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:07.594683    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f0ed547-c5bd-4a81-a5bf-91dcf8df131c" containerID="c28e8f7b9c4262f04a49039db8e1b4c55ea4090c9136e71a9fc02e9a1b680e61" exitCode=2
Sep 16 18:49:07.595189 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:07.594755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerDied","Data":"c28e8f7b9c4262f04a49039db8e1b4c55ea4090c9136e71a9fc02e9a1b680e61"}
Sep 16 18:49:07.595189 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:07.594800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d77d4d679-vzfd2" event={"ID":"0f0ed547-c5bd-4a81-a5bf-91dcf8df131c","Type":"ContainerStarted","Data":"0091ec444ebb9c98f442450b9a63fd87a4cc0fe8186b013ea0c54af79ccfdc9a"}
Sep 16 18:49:13.388868 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:13.388781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:13.458060 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:13.458016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:23.594128 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:23.594087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:49:23.596582 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:23.596551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dd2fed3-b044-4df8-9a04-0f1de72ac07c-metrics-certs\") pod \"network-metrics-daemon-mmtgl\" (UID: \"1dd2fed3-b044-4df8-9a04-0f1de72ac07c\") " pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:49:23.822474 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:23.822443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\""
Sep 16 18:49:23.829235 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:23.829212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmtgl"
Sep 16 18:49:24.098327 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:24.098295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mmtgl"]
Sep 16 18:49:24.101746 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:49:24.101719    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd2fed3_b044_4df8_9a04_0f1de72ac07c.slice/crio-5d59a9f9028c84923284961a26636741eccec2c4179a49231cb5baf465bfebb0 WatchSource:0}: Error finding container 5d59a9f9028c84923284961a26636741eccec2c4179a49231cb5baf465bfebb0: Status 404 returned error can't find the container with id 5d59a9f9028c84923284961a26636741eccec2c4179a49231cb5baf465bfebb0
Sep 16 18:49:24.653707 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:24.653664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmtgl" event={"ID":"1dd2fed3-b044-4df8-9a04-0f1de72ac07c","Type":"ContainerStarted","Data":"5d59a9f9028c84923284961a26636741eccec2c4179a49231cb5baf465bfebb0"}
Sep 16 18:49:25.659136 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:25.659092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmtgl" event={"ID":"1dd2fed3-b044-4df8-9a04-0f1de72ac07c","Type":"ContainerStarted","Data":"7840aeadabf912117013090c6262d9d932a69d67977d55e02b4b2b9609d0d848"}
Sep 16 18:49:25.659617 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:25.659178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmtgl" event={"ID":"1dd2fed3-b044-4df8-9a04-0f1de72ac07c","Type":"ContainerStarted","Data":"ffa253b81c211d35dbdebc7556d73f2b604fbe7425849904023f0e768d2ed36b"}
Sep 16 18:49:25.678811 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:25.678759    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mmtgl" podStartSLOduration=252.746894998 podStartE2EDuration="4m13.678744385s" totalImagesPullingTime="931.849387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:25.678094719 +0000 UTC m=+254.416670218" watchObservedRunningTime="2026-09-16 18:49:25.678744385 +0000 UTC m=+254.417319882"
Sep 16 18:49:25.788614 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:25.788582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:25.848372 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:25.848343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.219407 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:27.219375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.285643 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:27.285613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:33.900842 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:33.900789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:34.341623 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:34.341514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:42.322194 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:42.322157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:42.423265 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:42.423236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:42.530150 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:42.530120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:43.087501 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:43.087465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:53.411497 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:53.411461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:53.511595 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:53.511574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:53.613131 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:53.613109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:54.486748 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:54.486712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:54.559575 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:49:54.559547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.623657 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:11.623611    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:18.139097 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:18.139062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:18.240091 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:18.240067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.199166 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:49.199117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.264785 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:49.264752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:50.512075 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:50.512044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:50.585892 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:50:50.585865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:01.327572 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:01.327524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:01.482740 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:01.482708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:01.723674 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:01.723641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:01.824004 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:01.823976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:01.904050 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:01.904025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:13.431769 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:13.431686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:13.553121 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:51:13.553067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.286044 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:00.285995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:01.647620 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:01.647584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:27.745761 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:27.745724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:27.838848 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:27.838788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.494941 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:34.494801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.868574 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:34.868538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.105441 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.105405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.217594 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.217518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.289321 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.289291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.534240 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.534170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.639451 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.639422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.835731 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.835661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6962l"]
Sep 16 18:52:35.835881 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.835868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87221153-8517-4ad0-9df3-c252e1868753" containerName="registry"
Sep 16 18:52:35.835881 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.835879    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87221153-8517-4ad0-9df3-c252e1868753" containerName="registry"
Sep 16 18:52:35.835955 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.835941    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87221153-8517-4ad0-9df3-c252e1868753" containerName="registry"
Sep 16 18:52:35.838584 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.838568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:35.843685 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.843661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:52:35.843805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.843672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-z6rp6\""
Sep 16 18:52:35.843805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.843717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:52:35.843805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.843744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:52:35.843805 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.843767    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:52:35.856844 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.856805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6962l"]
Sep 16 18:52:35.881005 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.880983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.962778 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.962755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:35.998859 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.998837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-certificates\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:35.998960 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:35.998873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscx5\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-kube-api-access-zscx5\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.099516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.099452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-certificates\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.099516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.099489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscx5\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-kube-api-access-zscx5\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.102069 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.102048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-certificates\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.116834 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.116800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscx5\" (UniqueName: \"kubernetes.io/projected/5ee3dbf6-0e52-4840-917b-832b0b5a7965-kube-api-access-zscx5\") pod \"keda-admission-6c4d864b86-6962l\" (UID: \"5ee3dbf6-0e52-4840-917b-832b0b5a7965\") " pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.147746 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.147716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:36.222959 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.222932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:36.431642 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.431604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6962l"]
Sep 16 18:52:36.434067 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:52:36.434043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee3dbf6_0e52_4840_917b_832b0b5a7965.slice/crio-a65a2085c7c29c7cd152a35b0120166c79b306616186e72bfede2e00e20e6fba WatchSource:0}: Error finding container a65a2085c7c29c7cd152a35b0120166c79b306616186e72bfede2e00e20e6fba: Status 404 returned error can't find the container with id a65a2085c7c29c7cd152a35b0120166c79b306616186e72bfede2e00e20e6fba
Sep 16 18:52:36.435274 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.435258    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:36.448393 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.448364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:36.615499 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.615473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:36.716995 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:36.716924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:37.281400 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:37.281361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6962l" event={"ID":"5ee3dbf6-0e52-4840-917b-832b0b5a7965","Type":"ContainerStarted","Data":"a65a2085c7c29c7cd152a35b0120166c79b306616186e72bfede2e00e20e6fba"}
Sep 16 18:52:39.290386 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:39.290323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6962l" event={"ID":"5ee3dbf6-0e52-4840-917b-832b0b5a7965","Type":"ContainerStarted","Data":"d6c28b063292358bae4aa46a1208d0c28c0f0ba8e13027f124f67f14fba59540"}
Sep 16 18:52:39.290780 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:39.290450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:52:39.310893 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:39.310845    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-6962l" podStartSLOduration=2.119810543 podStartE2EDuration="4.31080322s" totalImagesPullingTime="2.190992677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:39.309534765 +0000 UTC m=+448.048110264" watchObservedRunningTime="2026-09-16 18:52:39.31080322 +0000 UTC m=+448.049378717"
Sep 16 18:52:42.307637 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:42.307605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:42.400258 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:52:42.400232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:00.296191 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:53:00.296145    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-6962l"
Sep 16 18:53:28.520442 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:53:28.520402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:28.621239 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:53:28.621214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.795291 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:53:55.795247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:57.541997 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:53:57.541955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.182983 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.182950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.261773 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.261739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.302175 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.302149    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2cx84"]
Sep 16 18:54:00.305021 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.305007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.308049 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.308030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:54:00.308706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.308692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:54:00.309577 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.309559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:54:00.309973 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.309954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hqw4s\""
Sep 16 18:54:00.318140 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.318122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2cx84"]
Sep 16 18:54:00.346883 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.346863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-data\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.346979 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.346916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7q5\" (UniqueName: \"kubernetes.io/projected/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-kube-api-access-9p7q5\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.380411 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.380385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.448203 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.448150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7q5\" (UniqueName: \"kubernetes.io/projected/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-kube-api-access-9p7q5\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.448203 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.448186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-data\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.448492 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.448476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-data\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.456950 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.456926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7q5\" (UniqueName: \"kubernetes.io/projected/1b76fedc-477b-4d7e-b9f7-6b777e8427c0-kube-api-access-9p7q5\") pod \"seaweedfs-64568bcd49-2cx84\" (UID: \"1b76fedc-477b-4d7e-b9f7-6b777e8427c0\") " pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.476176 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.476153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.613488 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.613455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:00.791703 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.791639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:00.882518 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.882460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2cx84"]
Sep 16 18:54:00.885092 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:54:00.885065    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b76fedc_477b_4d7e_b9f7_6b777e8427c0.slice/crio-30a7fd14dff072f5e12e83f02a85899ff1243e72812cc6094298bf46bac51075 WatchSource:0}: Error finding container 30a7fd14dff072f5e12e83f02a85899ff1243e72812cc6094298bf46bac51075: Status 404 returned error can't find the container with id 30a7fd14dff072f5e12e83f02a85899ff1243e72812cc6094298bf46bac51075
Sep 16 18:54:00.918493 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:00.918467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:01.017358 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:01.017335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:01.581500 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:01.581468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2cx84" event={"ID":"1b76fedc-477b-4d7e-b9f7-6b777e8427c0","Type":"ContainerStarted","Data":"30a7fd14dff072f5e12e83f02a85899ff1243e72812cc6094298bf46bac51075"}
Sep 16 18:54:02.982776 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:02.982735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:03.067692 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:03.067664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:03.590713 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:03.590628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2cx84" event={"ID":"1b76fedc-477b-4d7e-b9f7-6b777e8427c0","Type":"ContainerStarted","Data":"88cadc3ce8c76aa3ddfdff8892aee78f2bdaee8f1bf0e4daa41b6c52b679d68c"}
Sep 16 18:54:03.590890 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:03.590788    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:54:03.612548 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:03.612494    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2cx84" podStartSLOduration=1.170437393 podStartE2EDuration="3.612479987s" totalImagesPullingTime="2.442042594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:03.610886493 +0000 UTC m=+532.349461992" watchObservedRunningTime="2026-09-16 18:54:03.612479987 +0000 UTC m=+532.351055485"
Sep 16 18:54:09.595864 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:54:09.595752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2cx84"
Sep 16 18:55:17.976440 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:17.976407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.016455 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.016428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-gl8lw"]
Sep 16 18:55:18.020872 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.020842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.024187 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.024164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qv2bf\""
Sep 16 18:55:18.024318 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.024228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:55:18.024318 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.024282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:55:18.029851 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.029829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gl8lw"]
Sep 16 18:55:18.075701 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.075674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.134995 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.134974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjhw\" (UniqueName: \"kubernetes.io/projected/becdef2b-631f-4805-8359-deca750c8cb2-kube-api-access-krjhw\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.135084 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.135012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-metrics-certs\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.135084 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.135031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.219125 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.219054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.236249 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.236210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-metrics-certs\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.236249 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.236236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.236341 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.236263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjhw\" (UniqueName: \"kubernetes.io/projected/becdef2b-631f-4805-8359-deca750c8cb2-kube-api-access-krjhw\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.236384 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:55:18.236346    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:55:18.236419 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:55:18.236403    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert podName:becdef2b-631f-4805-8359-deca750c8cb2 nodeName:}" failed. No retries permitted until 2026-09-16 18:55:18.736389805 +0000 UTC m=+607.474965280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert") pod "odh-model-controller-7b966d7984-gl8lw" (UID: "becdef2b-631f-4805-8359-deca750c8cb2") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:55:18.238641 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.238616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-metrics-certs\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.249667 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.249648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjhw\" (UniqueName: \"kubernetes.io/projected/becdef2b-631f-4805-8359-deca750c8cb2-kube-api-access-krjhw\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.322241 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.322219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.423739 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.423713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:18.738627 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.738594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.741116 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.741094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becdef2b-631f-4805-8359-deca750c8cb2-cert\") pod \"odh-model-controller-7b966d7984-gl8lw\" (UID: \"becdef2b-631f-4805-8359-deca750c8cb2\") " pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:18.930620 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:18.930596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:19.203952 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:19.203894    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-gl8lw"]
Sep 16 18:55:19.206431 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:55:19.206405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecdef2b_631f_4805_8359_deca750c8cb2.slice/crio-96815958f60fe673a2867846a3ba3a55faf6b25d3f2638ab73f57c7d488d6f8e WatchSource:0}: Error finding container 96815958f60fe673a2867846a3ba3a55faf6b25d3f2638ab73f57c7d488d6f8e: Status 404 returned error can't find the container with id 96815958f60fe673a2867846a3ba3a55faf6b25d3f2638ab73f57c7d488d6f8e
Sep 16 18:55:19.240124 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:19.240102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:19.303715 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:19.303691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:19.852536 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:19.852497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gl8lw" event={"ID":"becdef2b-631f-4805-8359-deca750c8cb2","Type":"ContainerStarted","Data":"96815958f60fe673a2867846a3ba3a55faf6b25d3f2638ab73f57c7d488d6f8e"}
Sep 16 18:55:21.860461 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:21.860427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-gl8lw" event={"ID":"becdef2b-631f-4805-8359-deca750c8cb2","Type":"ContainerStarted","Data":"b68bbab0bab26bc515d1d92c2e025ed1aea570e765312a7e693500a8978f02e7"}
Sep 16 18:55:21.860923 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:21.860637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:21.883330 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:21.883287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-gl8lw" podStartSLOduration=2.477635336 podStartE2EDuration="4.883273627s" totalImagesPullingTime="2.405638291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:21.882991681 +0000 UTC m=+610.621567179" watchObservedRunningTime="2026-09-16 18:55:21.883273627 +0000 UTC m=+610.621849163"
Sep 16 18:55:22.711123 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:22.711075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:32.866706 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:32.866675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-gl8lw"
Sep 16 18:55:51.851782 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:55:51.851748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:34.352379 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:56:34.352337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:34.450167 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:56:34.450130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.448866 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:56:52.448811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:52.548481 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:56:52.548451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:04.742448 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.742373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:04.962147 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.962118    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:57:04.965423 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.965404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:04.968760 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.968732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:57:04.968890 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.968862    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:57:04.970179 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.970159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxpzh\""
Sep 16 18:57:04.970297 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.970209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dc809-predictor-serving-cert\""
Sep 16 18:57:04.970297 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.970219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-dc809-kube-rbac-proxy-sar-config\""
Sep 16 18:57:04.974538 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:04.974519    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:57:05.012192 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.012123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.012192 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.012179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.012332 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.012211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wct6g\" (UniqueName: \"kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.113262 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.113230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.113404 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.113269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wct6g\" (UniqueName: \"kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.113404 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.113298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.113972 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.113950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.115680 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.115662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.124620 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.124596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wct6g\" (UniqueName: \"kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g\") pod \"success-200-isvc-dc809-predictor-7df545f6f-cvr2x\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") " pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.276614 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.276554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:05.549869 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:05.549846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:57:05.552174 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:57:05.552144    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ac7ca3_a28b_44f6_981b_8e048fb5f02b.slice/crio-1466caea89fd38f361efb9faa93c89e6f640c1543cafb4cc2558c10a1abeb247 WatchSource:0}: Error finding container 1466caea89fd38f361efb9faa93c89e6f640c1543cafb4cc2558c10a1abeb247: Status 404 returned error can't find the container with id 1466caea89fd38f361efb9faa93c89e6f640c1543cafb4cc2558c10a1abeb247
Sep 16 18:57:06.214051 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:06.214016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerStarted","Data":"1466caea89fd38f361efb9faa93c89e6f640c1543cafb4cc2558c10a1abeb247"}
Sep 16 18:57:06.475018 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:06.474932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:18.266454 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:18.266380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerStarted","Data":"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"}
Sep 16 18:57:19.128136 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:19.128100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:20.276286 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:20.276251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerStarted","Data":"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"}
Sep 16 18:57:20.276687 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:20.276371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:20.297760 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:20.297714    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podStartSLOduration=2.349364057 podStartE2EDuration="16.297702824s" totalImagesPullingTime="13.948338767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:20.297330694 +0000 UTC m=+729.035906192" watchObservedRunningTime="2026-09-16 18:57:20.297702824 +0000 UTC m=+729.036278322"
Sep 16 18:57:21.279940 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:21.279906    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:21.281299 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:21.281274    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:57:22.283474 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:22.283427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:57:27.288128 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:27.288101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:57:27.288615 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:27.288587    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:57:30.013983 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:30.013947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:37.289142 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:37.289100    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:57:47.289344 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:47.289304    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:57:57.289210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:57:57.289165    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:58:07.289366 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:07.289336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:58:24.833254 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.833219    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:58:24.835471 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.835455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:24.839209 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.839187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-dc809-serving-cert\""
Sep 16 18:58:24.840152 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.840134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-dc809-kube-rbac-proxy-sar-config\""
Sep 16 18:58:24.850265 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.850240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:58:24.854861 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.854834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.859864 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.859845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:24.859939 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.859896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:24.961117 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.961092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:24.961223 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.961128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:24.961270 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:24.961223    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-dc809-serving-cert: secret "switch-graph-dc809-serving-cert" not found
Sep 16 18:58:24.961305 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:24.961294    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls podName:63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c nodeName:}" failed. No retries permitted until 2026-09-16 18:58:25.461279212 +0000 UTC m=+794.199854688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls") pod "switch-graph-dc809-8f5cc9c67-kfk4h" (UID: "63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c") : secret "switch-graph-dc809-serving-cert" not found
Sep 16 18:58:24.961687 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:24.961667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:25.213253 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:25.213219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:25.464429 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:25.464356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:25.466985 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:25.466960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") pod \"switch-graph-dc809-8f5cc9c67-kfk4h\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") " pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:25.745434 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:25.745371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:26.016276 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:26.016207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:58:26.018927 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:58:26.018890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d08ebb_7f01_4fe1_b1f7_6e8b1ae6ca9c.slice/crio-980580a5eabcd79282bbdcee01dffcd65b1fe98d7b375b7a96fed734b87ddf5e WatchSource:0}: Error finding container 980580a5eabcd79282bbdcee01dffcd65b1fe98d7b375b7a96fed734b87ddf5e: Status 404 returned error can't find the container with id 980580a5eabcd79282bbdcee01dffcd65b1fe98d7b375b7a96fed734b87ddf5e
Sep 16 18:58:26.020941 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:26.020923    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:26.505216 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:26.505184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" event={"ID":"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c","Type":"ContainerStarted","Data":"980580a5eabcd79282bbdcee01dffcd65b1fe98d7b375b7a96fed734b87ddf5e"}
Sep 16 18:58:28.514510 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:28.514478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" event={"ID":"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c","Type":"ContainerStarted","Data":"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"}
Sep 16 18:58:28.514858 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:28.514540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:28.534855 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:28.534798    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podStartSLOduration=2.15718356 podStartE2EDuration="4.534784543s" totalImagesPullingTime="2.377600983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:28.534606453 +0000 UTC m=+797.273181952" watchObservedRunningTime="2026-09-16 18:58:28.534784543 +0000 UTC m=+797.273360040"
Sep 16 18:58:34.523353 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.523275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:34.889525 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.889494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:58:34.889694 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.889662    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" containerID="cri-o://6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499" gracePeriod=30
Sep 16 18:58:34.975336 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.975305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:58:34.975560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.975535    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kube-rbac-proxy" containerID="cri-o://cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072" gracePeriod=30
Sep 16 18:58:34.975646 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:34.975520    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" containerID="cri-o://e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0" gracePeriod=30
Sep 16 18:58:35.097939 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.097912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 18:58:35.100861 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.100845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.103734 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.103711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9a5c2-kube-rbac-proxy-sar-config\""
Sep 16 18:58:35.103841 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.103747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9a5c2-predictor-serving-cert\""
Sep 16 18:58:35.111751 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.111730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 18:58:35.128066 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.128037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.128306 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.128288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hkx7\" (UniqueName: \"kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.128351 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.128321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.128414 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.128349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.229134 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.229066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.229134 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.229127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hkx7\" (UniqueName: \"kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.229354 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.229156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.229354 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:35.229244    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-serving-cert: secret "success-200-isvc-9a5c2-predictor-serving-cert" not found
Sep 16 18:58:35.229354 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:35.229301    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls podName:a7d922f7-f4bb-4330-9503-8c5f15c244be nodeName:}" failed. No retries permitted until 2026-09-16 18:58:35.729284788 +0000 UTC m=+804.467860270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls") pod "success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" (UID: "a7d922f7-f4bb-4330-9503-8c5f15c244be") : secret "success-200-isvc-9a5c2-predictor-serving-cert" not found
Sep 16 18:58:35.229959 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.229902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.231891 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.231872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.237743 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.237716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hkx7\" (UniqueName: \"kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.484663 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.484553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.540853 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.540810    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerID="cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072" exitCode=2
Sep 16 18:58:35.541180 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.540848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerDied","Data":"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"}
Sep 16 18:58:35.733133 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.733100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.735622 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.735567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") pod \"success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") " pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:35.745085 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:35.745041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.010997 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.010923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:36.283623 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.283597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 18:58:36.285426 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:58:36.285400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d922f7_f4bb_4330_9503_8c5f15c244be.slice/crio-7d5d89306f05cdbcc4fb098249434376cc5c615b997a22d236cec946aa0901a2 WatchSource:0}: Error finding container 7d5d89306f05cdbcc4fb098249434376cc5c615b997a22d236cec946aa0901a2: Status 404 returned error can't find the container with id 7d5d89306f05cdbcc4fb098249434376cc5c615b997a22d236cec946aa0901a2
Sep 16 18:58:36.547894 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.547789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerStarted","Data":"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"}
Sep 16 18:58:36.547894 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.547882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerStarted","Data":"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"}
Sep 16 18:58:36.548330 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.547917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerStarted","Data":"7d5d89306f05cdbcc4fb098249434376cc5c615b997a22d236cec946aa0901a2"}
Sep 16 18:58:36.548330 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.547961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:36.568089 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:36.568045    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podStartSLOduration=1.568030232 podStartE2EDuration="1.568030232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:36.567279932 +0000 UTC m=+805.305855431" watchObservedRunningTime="2026-09-16 18:58:36.568030232 +0000 UTC m=+805.306605729"
Sep 16 18:58:37.284238 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.284192    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
Sep 16 18:58:37.288525 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.288496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 16 18:58:37.553188 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.553152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:37.554671 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.554640    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:58:37.712499 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.712476    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:58:37.748225 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.748200    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls\") pod \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") "
Sep 16 18:58:37.748359 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.748281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config\") pod \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") "
Sep 16 18:58:37.748359 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.748306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wct6g\" (UniqueName: \"kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g\") pod \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\" (UID: \"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b\") "
Sep 16 18:58:37.748731 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.748691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" (UID: "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:37.750508 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.750480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" (UID: "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:37.750613 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.750528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g" (OuterVolumeSpecName: "kube-api-access-wct6g") pod "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" (UID: "d1ac7ca3-a28b-44f6-981b-8e048fb5f02b"). InnerVolumeSpecName "kube-api-access-wct6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:37.849076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.849003    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:58:37.849076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.849029    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wct6g\" (UniqueName: \"kubernetes.io/projected/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-kube-api-access-wct6g\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:58:37.849076 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:37.849040    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:58:38.558748 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.558710    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerID="e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0" exitCode=0
Sep 16 18:58:38.559168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.558771    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"
Sep 16 18:58:38.559168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.558796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerDied","Data":"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"}
Sep 16 18:58:38.559168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.558882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x" event={"ID":"d1ac7ca3-a28b-44f6-981b-8e048fb5f02b","Type":"ContainerDied","Data":"1466caea89fd38f361efb9faa93c89e6f640c1543cafb4cc2558c10a1abeb247"}
Sep 16 18:58:38.559168 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.558910    2563 scope.go:117] "RemoveContainer" containerID="cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"
Sep 16 18:58:38.559368 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.559300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:58:38.567947 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.567931    2563 scope.go:117] "RemoveContainer" containerID="e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"
Sep 16 18:58:38.575032 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.575010    2563 scope.go:117] "RemoveContainer" containerID="cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"
Sep 16 18:58:38.575301 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:38.575283    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072\": container with ID starting with cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072 not found: ID does not exist" containerID="cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"
Sep 16 18:58:38.575349 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.575310    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072"} err="failed to get container status \"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072\": rpc error: code = NotFound desc = could not find container \"cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072\": container with ID starting with cf245283466aec68f933ee3d1cbb316034f266a7397a69027c944a23077d8072 not found: ID does not exist"
Sep 16 18:58:38.575349 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.575327    2563 scope.go:117] "RemoveContainer" containerID="e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"
Sep 16 18:58:38.575601 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:58:38.575580    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0\": container with ID starting with e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0 not found: ID does not exist" containerID="e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"
Sep 16 18:58:38.575649 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.575608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0"} err="failed to get container status \"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0\": rpc error: code = NotFound desc = could not find container \"e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0\": container with ID starting with e95609f791a9b77d0b6426e572fb3b74426f91497690e25411c30755bc6500b0 not found: ID does not exist"
Sep 16 18:58:38.591929 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.591907    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:58:38.596218 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:38.596200    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-dc809-predictor-7df545f6f-cvr2x"]
Sep 16 18:58:39.521658 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:39.521622    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:39.722427 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:39.722398    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" path="/var/lib/kubelet/pods/d1ac7ca3-a28b-44f6-981b-8e048fb5f02b/volumes"
Sep 16 18:58:43.564854 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:43.564805    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:58:43.565252 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:43.565222    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:58:44.521926 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:44.521882    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:49.521335 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:49.521299    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:49.521664 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:49.521405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:58:53.565981 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:53.565942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:58:54.521891 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:54.521837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:59.522475 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:58:59.522419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:03.565976 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:03.565936    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:59:04.521526 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:04.521488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:05.523010 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.522987    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:59:05.631741 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.631714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") pod \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") "
Sep 16 18:59:05.631933 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.631786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle\") pod \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\" (UID: \"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c\") "
Sep 16 18:59:05.632183 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.632156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" (UID: "63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:05.633859 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.633837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" (UID: "63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:05.656949 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.656892    2563 generic.go:356] "Generic (PLEG): container finished" podID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerID="6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499" exitCode=0
Sep 16 18:59:05.656949 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.656936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"
Sep 16 18:59:05.657057 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.656970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" event={"ID":"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c","Type":"ContainerDied","Data":"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"}
Sep 16 18:59:05.657057 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.657039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h" event={"ID":"63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c","Type":"ContainerDied","Data":"980580a5eabcd79282bbdcee01dffcd65b1fe98d7b375b7a96fed734b87ddf5e"}
Sep 16 18:59:05.657120 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.657062    2563 scope.go:117] "RemoveContainer" containerID="6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"
Sep 16 18:59:05.665211 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.665191    2563 scope.go:117] "RemoveContainer" containerID="6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"
Sep 16 18:59:05.665472 ip-10-0-137-226 kubenswrapper[2563]: E0916 18:59:05.665452    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499\": container with ID starting with 6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499 not found: ID does not exist" containerID="6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"
Sep 16 18:59:05.665528 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.665481    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499"} err="failed to get container status \"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499\": rpc error: code = NotFound desc = could not find container \"6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499\": container with ID starting with 6359e54aa59940353f03e8f26e1039beaf3b33662a2be98049113ca6497ae499 not found: ID does not exist"
Sep 16 18:59:05.679238 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.679218    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:59:05.684496 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.684477    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-dc809-8f5cc9c67-kfk4h"]
Sep 16 18:59:05.722062 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.722036    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" path="/var/lib/kubelet/pods/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c/volumes"
Sep 16 18:59:05.732492 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.732476    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:59:05.732560 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:05.732494    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 18:59:07.088936 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:07.088904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:07.190966 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:07.190939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:13.565609 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:13.565573    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 16 18:59:18.162915 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:18.162878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:18.276986 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:18.276952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.565972 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:23.565942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 18:59:24.760320 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760285    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760598    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kube-rbac-proxy"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kube-rbac-proxy"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kube-rbac-proxy"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760684    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ac7ca3-a28b-44f6-981b-8e048fb5f02b" containerName="kserve-container"
Sep 16 18:59:24.760696 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.760691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d08ebb-7f01-4fe1-b1f7-6e8b1ae6ca9c" containerName="switch-graph-dc809"
Sep 16 18:59:24.764784 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.764764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.767596 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.767573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 16 18:59:24.767732 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.767687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 16 18:59:24.773320 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.773296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 18:59:24.804934 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.804897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:24.856682 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.856662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.856770 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.856692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.957210 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.957184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.957306 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.957215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.957871 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.957841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.959653 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.959629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls\") pod \"model-chainer-6cd7f458db-8tkvv\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") " pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:24.998419 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:24.998393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:25.075702 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.075642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:25.351310 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.351238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 18:59:25.354890 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:59:25.354859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958bc847_30e5_4e63_8bbe_71ec1d16dd90.slice/crio-da083f6fc7e17ad8b2c236a09408de727f3fbb588fc7ab17ca357c9a15dad226 WatchSource:0}: Error finding container da083f6fc7e17ad8b2c236a09408de727f3fbb588fc7ab17ca357c9a15dad226: Status 404 returned error can't find the container with id da083f6fc7e17ad8b2c236a09408de727f3fbb588fc7ab17ca357c9a15dad226
Sep 16 18:59:25.733215 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.733181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" event={"ID":"958bc847-30e5-4e63-8bbe-71ec1d16dd90","Type":"ContainerStarted","Data":"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"}
Sep 16 18:59:25.733362 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.733238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" event={"ID":"958bc847-30e5-4e63-8bbe-71ec1d16dd90","Type":"ContainerStarted","Data":"da083f6fc7e17ad8b2c236a09408de727f3fbb588fc7ab17ca357c9a15dad226"}
Sep 16 18:59:25.733362 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.733329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:25.751449 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:25.751395    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podStartSLOduration=1.7513795760000002 podStartE2EDuration="1.751379576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:25.750916678 +0000 UTC m=+854.489492178" watchObservedRunningTime="2026-09-16 18:59:25.751379576 +0000 UTC m=+854.489955111"
Sep 16 18:59:31.742833 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:31.742791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:34.866138 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:34.866106    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 18:59:34.866644 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:34.866273    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" containerID="cri-o://4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c" gracePeriod=30
Sep 16 18:59:34.928347 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:34.928322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.052251 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.052223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.115707 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.115675    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 18:59:35.118729 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.118688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.122968 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.122949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-9a5c2-serving-cert\""
Sep 16 18:59:35.123071 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.122987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-9a5c2-kube-rbac-proxy-sar-config\""
Sep 16 18:59:35.132460 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.132437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 18:59:35.135619 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.135600    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 18:59:35.138516 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.138501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.141487 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.141463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a895e-kube-rbac-proxy-sar-config\""
Sep 16 18:59:35.142094 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.142079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a895e-predictor-serving-cert\""
Sep 16 18:59:35.150730 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.150711    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 18:59:35.184308 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.184287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.222113 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.222091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzg4g\" (UniqueName: \"kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.222235 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.222147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.222235 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.222176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.222343 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.222240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.222343 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.222270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.313156 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.313130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.323169 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.323239 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzg4g\" (UniqueName: \"kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.323239 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.323239 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.323351 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.323863 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.323963 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.323938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.325533 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.325514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls\") pod \"switch-graph-9a5c2-755f799d8b-t59th\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") " pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.325697 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.325676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.332113 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.332090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzg4g\" (UniqueName: \"kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g\") pod \"success-200-isvc-a895e-predictor-79cfc466b9-vkzsw\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") " pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.429174 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.429126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.445357 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.445331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.448073 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.448051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:35.701613 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.701588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 18:59:35.703883 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:59:35.703860    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf749f10b_5983_4dde_ae73_d31c25d8051e.slice/crio-3ab7ad217b6d97129c550cab30901447cbf1527033f5115c68807be53d87b05c WatchSource:0}: Error finding container 3ab7ad217b6d97129c550cab30901447cbf1527033f5115c68807be53d87b05c: Status 404 returned error can't find the container with id 3ab7ad217b6d97129c550cab30901447cbf1527033f5115c68807be53d87b05c
Sep 16 18:59:35.723612 ip-10-0-137-226 kubenswrapper[2563]: W0916 18:59:35.723589    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c11b683_438e_445f_91f5_733cc6ff934a.slice/crio-39e83ae2a6170c69509ce98533838012523a533994277f2d8d6429c2c05d1633 WatchSource:0}: Error finding container 39e83ae2a6170c69509ce98533838012523a533994277f2d8d6429c2c05d1633: Status 404 returned error can't find the container with id 39e83ae2a6170c69509ce98533838012523a533994277f2d8d6429c2c05d1633
Sep 16 18:59:35.724918 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.724896    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 18:59:35.771878 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.771770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerStarted","Data":"39e83ae2a6170c69509ce98533838012523a533994277f2d8d6429c2c05d1633"}
Sep 16 18:59:35.773550 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.773524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" event={"ID":"f749f10b-5983-4dde-ae73-d31c25d8051e","Type":"ContainerStarted","Data":"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"}
Sep 16 18:59:35.773648 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.773575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" event={"ID":"f749f10b-5983-4dde-ae73-d31c25d8051e","Type":"ContainerStarted","Data":"3ab7ad217b6d97129c550cab30901447cbf1527033f5115c68807be53d87b05c"}
Sep 16 18:59:35.773707 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.773662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:35.792332 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:35.792283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podStartSLOduration=0.792262507 podStartE2EDuration="792.262507ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:35.791059109 +0000 UTC m=+864.529634609" watchObservedRunningTime="2026-09-16 18:59:35.792262507 +0000 UTC m=+864.530838007"
Sep 16 18:59:36.742329 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:36.742292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:36.778914 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:36.778872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerStarted","Data":"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"}
Sep 16 18:59:36.779093 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:36.778947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerStarted","Data":"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"}
Sep 16 18:59:36.799917 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:36.799874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podStartSLOduration=1.7998606700000002 podStartE2EDuration="1.79986067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:36.79869107 +0000 UTC m=+865.537266596" watchObservedRunningTime="2026-09-16 18:59:36.79986067 +0000 UTC m=+865.538436167"
Sep 16 18:59:37.782879 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:37.782846    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:37.783239 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:37.782913    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:37.784158 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:37.784130    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:38.786617 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:38.786574    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:41.741746 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:41.741697    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:41.784563 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:41.784538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 18:59:43.792010 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:43.791907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 18:59:43.792421 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:43.792346    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:46.743828 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:46.743777    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:46.744248 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:46.743899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 18:59:51.740997 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:51.740958    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:53.793357 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:53.793312    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 18:59:56.741298 ip-10-0-137-226 kubenswrapper[2563]: I0916 18:59:56.741261    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:01.741621 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:01.741574    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:03.793035 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:03.792994    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:05.050567 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.050541    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 19:00:05.133503 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.133443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle\") pod \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") "
Sep 16 19:00:05.133634 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.133512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls\") pod \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\" (UID: \"958bc847-30e5-4e63-8bbe-71ec1d16dd90\") "
Sep 16 19:00:05.133836 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.133795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "958bc847-30e5-4e63-8bbe-71ec1d16dd90" (UID: "958bc847-30e5-4e63-8bbe-71ec1d16dd90"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:05.135559 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.135532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "958bc847-30e5-4e63-8bbe-71ec1d16dd90" (UID: "958bc847-30e5-4e63-8bbe-71ec1d16dd90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:05.234697 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.234675    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/958bc847-30e5-4e63-8bbe-71ec1d16dd90-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:00:05.234697 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.234699    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/958bc847-30e5-4e63-8bbe-71ec1d16dd90-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:00:05.883174 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.883135    2563 generic.go:356] "Generic (PLEG): container finished" podID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerID="4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c" exitCode=0
Sep 16 19:00:05.883316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.883198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"
Sep 16 19:00:05.883316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.883204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" event={"ID":"958bc847-30e5-4e63-8bbe-71ec1d16dd90","Type":"ContainerDied","Data":"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"}
Sep 16 19:00:05.883316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.883254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv" event={"ID":"958bc847-30e5-4e63-8bbe-71ec1d16dd90","Type":"ContainerDied","Data":"da083f6fc7e17ad8b2c236a09408de727f3fbb588fc7ab17ca357c9a15dad226"}
Sep 16 19:00:05.883316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.883285    2563 scope.go:117] "RemoveContainer" containerID="4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"
Sep 16 19:00:05.891185 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.891165    2563 scope.go:117] "RemoveContainer" containerID="4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"
Sep 16 19:00:05.891532 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:00:05.891494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c\": container with ID starting with 4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c not found: ID does not exist" containerID="4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"
Sep 16 19:00:05.891582 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.891527    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c"} err="failed to get container status \"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c\": rpc error: code = NotFound desc = could not find container \"4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c\": container with ID starting with 4702a79077b6af33928e682fbc91cd980ad78745355bcc88cd0cbbb1afee147c not found: ID does not exist"
Sep 16 19:00:05.901776 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.901751    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 19:00:05.905479 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:05.905458    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6cd7f458db-8tkvv"]
Sep 16 19:00:07.722741 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:07.722692    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" path="/var/lib/kubelet/pods/958bc847-30e5-4e63-8bbe-71ec1d16dd90/volumes"
Sep 16 19:00:12.080597 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:12.080522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:13.793062 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:13.793016    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 16 19:00:23.793059 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:23.793022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 19:00:35.090107 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.090073    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:00:35.090486 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.090355    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer"
Sep 16 19:00:35.090486 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.090366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer"
Sep 16 19:00:35.090486 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.090418    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="958bc847-30e5-4e63-8bbe-71ec1d16dd90" containerName="model-chainer"
Sep 16 19:00:35.093028 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.093013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.096366 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.096347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a895e-serving-cert\""
Sep 16 19:00:35.097268 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.097250    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a895e-kube-rbac-proxy-sar-config\""
Sep 16 19:00:35.105239 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.105214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:00:35.231312 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.231288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.231420 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.231326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.269720 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.269698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.331845 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.331801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.331927 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.331861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.332408 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.332391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.334165 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.334147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls\") pod \"sequence-graph-a895e-569fdff559-p2sc5\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") " pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.403162 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.403143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:35.416939 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.416921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.685997 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.685975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:00:35.688227 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:00:35.688204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3911f4_2ade_4427_92f0_7f041c973e61.slice/crio-f97d80de42720642502dd2d18085685e430f183de02bf10062216e9639854e43 WatchSource:0}: Error finding container f97d80de42720642502dd2d18085685e430f183de02bf10062216e9639854e43: Status 404 returned error can't find the container with id f97d80de42720642502dd2d18085685e430f183de02bf10062216e9639854e43
Sep 16 19:00:35.990433 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.990357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" event={"ID":"da3911f4-2ade-4427-92f0-7f041c973e61","Type":"ContainerStarted","Data":"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"}
Sep 16 19:00:35.990433 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.990413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" event={"ID":"da3911f4-2ade-4427-92f0-7f041c973e61","Type":"ContainerStarted","Data":"f97d80de42720642502dd2d18085685e430f183de02bf10062216e9639854e43"}
Sep 16 19:00:35.990600 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:35.990501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:00:36.012030 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:36.011989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podStartSLOduration=1.011974348 podStartE2EDuration="1.011974348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:36.010228126 +0000 UTC m=+924.748803624" watchObservedRunningTime="2026-09-16 19:00:36.011974348 +0000 UTC m=+924.750549891"
Sep 16 19:00:41.999075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:00:41.999044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:02:08.747182 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:02:08.747131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:26.478099 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:03:26.477987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:49.819370 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:49.819293    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 19:07:49.821654 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:49.819497    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" containerID="cri-o://d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75" gracePeriod=30
Sep 16 19:07:50.062001 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.061975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:50.169599 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.169572    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 19:07:50.169838 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.169798    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container" containerID="cri-o://5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e" gracePeriod=30
Sep 16 19:07:50.169897 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.169867    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kube-rbac-proxy" containerID="cri-o://80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c" gracePeriod=30
Sep 16 19:07:50.233923 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.233890    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:07:50.237034 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.237017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.240609 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.240591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-443e6-predictor-serving-cert\""
Sep 16 19:07:50.241080 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.241062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-443e6-kube-rbac-proxy-sar-config\""
Sep 16 19:07:50.251583 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.251561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:07:50.292214 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.292191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.292316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.292237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.292316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.292259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvmg\" (UniqueName: \"kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.392967 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.392940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.393078 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.392985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.393078 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.393004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvmg\" (UniqueName: \"kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.393179 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:07:50.393090    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-443e6-predictor-serving-cert: secret "success-200-isvc-443e6-predictor-serving-cert" not found
Sep 16 19:07:50.393179 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:07:50.393153    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls podName:8100d59c-40c8-4523-8dd5-e31a6c5cb54b nodeName:}" failed. No retries permitted until 2026-09-16 19:07:50.893134216 +0000 UTC m=+1359.631709692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls") pod "success-200-isvc-443e6-predictor-c76bc877c-x6z66" (UID: "8100d59c-40c8-4523-8dd5-e31a6c5cb54b") : secret "success-200-isvc-443e6-predictor-serving-cert" not found
Sep 16 19:07:50.393569 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.393552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.405678 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.405643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvmg\" (UniqueName: \"kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.588838 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.588757    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerID="80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c" exitCode=2
Sep 16 19:07:50.588838 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.588824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerDied","Data":"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"}
Sep 16 19:07:50.895916 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.895878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:50.898223 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:50.898202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") pod \"success-200-isvc-443e6-predictor-c76bc877c-x6z66\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") " pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:51.146465 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.146386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:51.418104 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.418087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:07:51.420238 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:07:51.420213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8100d59c_40c8_4523_8dd5_e31a6c5cb54b.slice/crio-c073e8d60f74eb4bf24e44162e7c6529f5567d6c9728b77cf1757a2c6e511ff1 WatchSource:0}: Error finding container c073e8d60f74eb4bf24e44162e7c6529f5567d6c9728b77cf1757a2c6e511ff1: Status 404 returned error can't find the container with id c073e8d60f74eb4bf24e44162e7c6529f5567d6c9728b77cf1757a2c6e511ff1
Sep 16 19:07:51.594471 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.594441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerStarted","Data":"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"}
Sep 16 19:07:51.594630 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.594507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerStarted","Data":"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"}
Sep 16 19:07:51.594630 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.594612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:51.594706 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.594645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerStarted","Data":"c073e8d60f74eb4bf24e44162e7c6529f5567d6c9728b77cf1757a2c6e511ff1"}
Sep 16 19:07:51.616719 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.616674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podStartSLOduration=1.6166595780000002 podStartE2EDuration="1.616659578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:51.61550386 +0000 UTC m=+1360.354079359" watchObservedRunningTime="2026-09-16 19:07:51.616659578 +0000 UTC m=+1360.355235075"
Sep 16 19:07:51.651463 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.651441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:51.783146 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:51.783081    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:52.598589 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:52.598556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:52.599910 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:52.599877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:07:52.996991 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:52.996965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 19:07:53.114801 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.114776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hkx7\" (UniqueName: \"kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7\") pod \"a7d922f7-f4bb-4330-9503-8c5f15c244be\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") "
Sep 16 19:07:53.114942 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.114830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config\") pod \"a7d922f7-f4bb-4330-9503-8c5f15c244be\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") "
Sep 16 19:07:53.114942 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.114856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") pod \"a7d922f7-f4bb-4330-9503-8c5f15c244be\" (UID: \"a7d922f7-f4bb-4330-9503-8c5f15c244be\") "
Sep 16 19:07:53.115200 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.115177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7d922f7-f4bb-4330-9503-8c5f15c244be" (UID: "a7d922f7-f4bb-4330-9503-8c5f15c244be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:53.116937 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.116916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7" (OuterVolumeSpecName: "kube-api-access-5hkx7") pod "a7d922f7-f4bb-4330-9503-8c5f15c244be" (UID: "a7d922f7-f4bb-4330-9503-8c5f15c244be"). InnerVolumeSpecName "kube-api-access-5hkx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:53.117038 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.116950    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7d922f7-f4bb-4330-9503-8c5f15c244be" (UID: "a7d922f7-f4bb-4330-9503-8c5f15c244be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:53.215843 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.215806    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hkx7\" (UniqueName: \"kubernetes.io/projected/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-api-access-5hkx7\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:07:53.215843 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.215840    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7d922f7-f4bb-4330-9503-8c5f15c244be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:07:53.215969 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.215852    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7d922f7-f4bb-4330-9503-8c5f15c244be-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:07:53.603159 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603082    2563 generic.go:356] "Generic (PLEG): container finished" podID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerID="5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e" exitCode=0
Sep 16 19:07:53.603159 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603134    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"
Sep 16 19:07:53.603559 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerDied","Data":"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"}
Sep 16 19:07:53.603559 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb" event={"ID":"a7d922f7-f4bb-4330-9503-8c5f15c244be","Type":"ContainerDied","Data":"7d5d89306f05cdbcc4fb098249434376cc5c615b997a22d236cec946aa0901a2"}
Sep 16 19:07:53.603559 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603269    2563 scope.go:117] "RemoveContainer" containerID="80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"
Sep 16 19:07:53.603743 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.603700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:07:53.612073 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.612057    2563 scope.go:117] "RemoveContainer" containerID="5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"
Sep 16 19:07:53.619247 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.619228    2563 scope.go:117] "RemoveContainer" containerID="80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"
Sep 16 19:07:53.619500 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:07:53.619482    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c\": container with ID starting with 80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c not found: ID does not exist" containerID="80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"
Sep 16 19:07:53.619564 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.619513    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c"} err="failed to get container status \"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c\": rpc error: code = NotFound desc = could not find container \"80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c\": container with ID starting with 80b26e393aa16f51054dd2bd9b8e10d65fb4119c7e432e743781d28c6102178c not found: ID does not exist"
Sep 16 19:07:53.619564 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.619536    2563 scope.go:117] "RemoveContainer" containerID="5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"
Sep 16 19:07:53.619826 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:07:53.619794    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e\": container with ID starting with 5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e not found: ID does not exist" containerID="5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"
Sep 16 19:07:53.619874 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.619840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e"} err="failed to get container status \"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e\": rpc error: code = NotFound desc = could not find container \"5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e\": container with ID starting with 5920d372d230c6abec599f610c7f3e5806b0be8f15d59992068e1038fac4560e not found: ID does not exist"
Sep 16 19:07:53.625565 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.625542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 19:07:53.631044 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.631023    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a5c2-predictor-79ccbb56df-zsrwb"]
Sep 16 19:07:53.722274 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:53.722254    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" path="/var/lib/kubelet/pods/a7d922f7-f4bb-4330-9503-8c5f15c244be/volumes"
Sep 16 19:07:56.783406 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:56.783368    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:07:58.608146 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:58.608115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:07:58.608612 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:07:58.608581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:08:01.782906 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:01.782862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:01.783307 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:01.782962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 19:08:06.783279 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:06.783238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:08.608712 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:08.608672    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:08:11.783119 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:11.783085    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:16.782905 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:16.782867    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:18.609561 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:18.609520    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:08:19.988012 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:19.987987    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 19:08:20.095234 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.095176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls\") pod \"f749f10b-5983-4dde-ae73-d31c25d8051e\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") "
Sep 16 19:08:20.095234 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.095210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle\") pod \"f749f10b-5983-4dde-ae73-d31c25d8051e\" (UID: \"f749f10b-5983-4dde-ae73-d31c25d8051e\") "
Sep 16 19:08:20.095581 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.095558    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f749f10b-5983-4dde-ae73-d31c25d8051e" (UID: "f749f10b-5983-4dde-ae73-d31c25d8051e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:20.097170 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.097147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f749f10b-5983-4dde-ae73-d31c25d8051e" (UID: "f749f10b-5983-4dde-ae73-d31c25d8051e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:20.196102 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.196079    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f749f10b-5983-4dde-ae73-d31c25d8051e-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:08:20.196102 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.196101    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f749f10b-5983-4dde-ae73-d31c25d8051e-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:08:20.703309 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.703289    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"
Sep 16 19:08:20.703435 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.703308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" event={"ID":"f749f10b-5983-4dde-ae73-d31c25d8051e","Type":"ContainerDied","Data":"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"}
Sep 16 19:08:20.703435 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.703358    2563 scope.go:117] "RemoveContainer" containerID="d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"
Sep 16 19:08:20.703435 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.703280    2563 generic.go:356] "Generic (PLEG): container finished" podID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerID="d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75" exitCode=137
Sep 16 19:08:20.703558 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.703493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th" event={"ID":"f749f10b-5983-4dde-ae73-d31c25d8051e","Type":"ContainerDied","Data":"3ab7ad217b6d97129c550cab30901447cbf1527033f5115c68807be53d87b05c"}
Sep 16 19:08:20.712289 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.712271    2563 scope.go:117] "RemoveContainer" containerID="d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"
Sep 16 19:08:20.712575 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:08:20.712557    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75\": container with ID starting with d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75 not found: ID does not exist" containerID="d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"
Sep 16 19:08:20.712615 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.712583    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75"} err="failed to get container status \"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75\": rpc error: code = NotFound desc = could not find container \"d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75\": container with ID starting with d30bb137bd7a8a953ea8781ad2a8a87983e44951b597a2110f4d76d23cf10b75 not found: ID does not exist"
Sep 16 19:08:20.725339 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.725313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 19:08:20.729039 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:20.729019    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-9a5c2-755f799d8b-t59th"]
Sep 16 19:08:21.722402 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:21.722371    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" path="/var/lib/kubelet/pods/f749f10b-5983-4dde-ae73-d31c25d8051e/volumes"
Sep 16 19:08:28.609508 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:28.609470    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 16 19:08:38.610107 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:38.610070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:08:49.739140 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.739111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:08:49.739532 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.739281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" containerID="cri-o://411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114" gracePeriod=30
Sep 16 19:08:49.794628 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.794595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:49.824456 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.824424    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 19:08:49.824709 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.824679    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container" containerID="cri-o://26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb" gracePeriod=30
Sep 16 19:08:49.824811 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.824756    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kube-rbac-proxy" containerID="cri-o://3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0" gracePeriod=30
Sep 16 19:08:49.894720 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.894690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:49.964772 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.964743    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:08:49.965040 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kube-rbac-proxy"
Sep 16 19:08:49.965083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965041    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kube-rbac-proxy"
Sep 16 19:08:49.965083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965055    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2"
Sep 16 19:08:49.965083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965061    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2"
Sep 16 19:08:49.965083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965067    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container"
Sep 16 19:08:49.965083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965073    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container"
Sep 16 19:08:49.965233 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kserve-container"
Sep 16 19:08:49.965233 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d922f7-f4bb-4330-9503-8c5f15c244be" containerName="kube-rbac-proxy"
Sep 16 19:08:49.965233 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.965133    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f749f10b-5983-4dde-ae73-d31c25d8051e" containerName="switch-graph-9a5c2"
Sep 16 19:08:49.968139 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.968118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:49.970959 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.970937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9450b-kube-rbac-proxy-sar-config\""
Sep 16 19:08:49.971096 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.971072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9450b-predictor-serving-cert\""
Sep 16 19:08:49.979180 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.979159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:08:49.982628 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:49.982590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:50.082652 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.082557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:50.099864 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.099834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.099995 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.099899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.099995 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.099929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzv2\" (UniqueName: \"kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.153075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.153038    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:08:50.156201 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.156180    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.159451 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.159426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-443e6-kube-rbac-proxy-sar-config\""
Sep 16 19:08:50.159788 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.159773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-443e6-serving-cert\""
Sep 16 19:08:50.165314 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.165289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:08:50.183022 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.182988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:50.201068 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.201038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.201165 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.201093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.201225 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:08:50.201173    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9450b-predictor-serving-cert: secret "success-200-isvc-9450b-predictor-serving-cert" not found
Sep 16 19:08:50.201225 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.201198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzv2\" (UniqueName: \"kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.201329 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:08:50.201237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls podName:2e236bb2-a567-40ef-8a2b-dcaea42291c3 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:50.701218035 +0000 UTC m=+1419.439793531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls") pod "success-200-isvc-9450b-predictor-67ccb68c65-x62p8" (UID: "2e236bb2-a567-40ef-8a2b-dcaea42291c3") : secret "success-200-isvc-9450b-predictor-serving-cert" not found
Sep 16 19:08:50.201691 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.201670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.210325 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.210299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzv2\" (UniqueName: \"kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.302028 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.301982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.302186 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.302109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.335754 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.335657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:50.403100 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.403079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.403233 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.403116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.403726 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.403703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.405463 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.405444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls\") pod \"ensemble-graph-443e6-7c548ddd5f-b4hgw\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") " pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.466370 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.466348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.476316 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.476295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:50.706088 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.706055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.708327 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.708309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") pod \"success-200-isvc-9450b-predictor-67ccb68c65-x62p8\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") " pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:50.735481 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.735461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:08:50.737266 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:08:50.737242    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5946f17_a43a_473c_b210_ce01616020bb.slice/crio-be54de767f77fe57c41d75a401365e75129f16fff0d4be1e49c4c21cfc963867 WatchSource:0}: Error finding container be54de767f77fe57c41d75a401365e75129f16fff0d4be1e49c4c21cfc963867: Status 404 returned error can't find the container with id be54de767f77fe57c41d75a401365e75129f16fff0d4be1e49c4c21cfc963867
Sep 16 19:08:50.813469 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.813435    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c11b683-438e-445f-91f5-733cc6ff934a" containerID="3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0" exitCode=2
Sep 16 19:08:50.813938 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.813477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerDied","Data":"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"}
Sep 16 19:08:50.815128 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.815109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" event={"ID":"d5946f17-a43a-473c-b210-ce01616020bb","Type":"ContainerStarted","Data":"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"}
Sep 16 19:08:50.815228 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.815148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" event={"ID":"d5946f17-a43a-473c-b210-ce01616020bb","Type":"ContainerStarted","Data":"be54de767f77fe57c41d75a401365e75129f16fff0d4be1e49c4c21cfc963867"}
Sep 16 19:08:50.815306 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.815290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:50.835499 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.835454    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podStartSLOduration=0.835440221 podStartE2EDuration="835.440221ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:50.834448007 +0000 UTC m=+1419.573023506" watchObservedRunningTime="2026-09-16 19:08:50.835440221 +0000 UTC m=+1419.574015719"
Sep 16 19:08:50.877756 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:50.877732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:51.149646 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.149587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:08:51.152218 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:08:51.152188    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e236bb2_a567_40ef_8a2b_dcaea42291c3.slice/crio-31c769b84262667d41ae15483d02c6b0f04d26c6b94e5b9ddd28d8a717d97ebd WatchSource:0}: Error finding container 31c769b84262667d41ae15483d02c6b0f04d26c6b94e5b9ddd28d8a717d97ebd: Status 404 returned error can't find the container with id 31c769b84262667d41ae15483d02c6b0f04d26c6b94e5b9ddd28d8a717d97ebd
Sep 16 19:08:51.821275 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.821237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerStarted","Data":"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"}
Sep 16 19:08:51.821748 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.821308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerStarted","Data":"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"}
Sep 16 19:08:51.821748 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.821341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerStarted","Data":"31c769b84262667d41ae15483d02c6b0f04d26c6b94e5b9ddd28d8a717d97ebd"}
Sep 16 19:08:51.821748 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.821524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:51.821748 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.821564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:51.822908 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.822877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:08:51.841385 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.841348    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podStartSLOduration=2.841333524 podStartE2EDuration="2.841333524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:51.839890429 +0000 UTC m=+1420.578465928" watchObservedRunningTime="2026-09-16 19:08:51.841333524 +0000 UTC m=+1420.579909025"
Sep 16 19:08:51.997299 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:51.997266    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:52.766247 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.766222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 19:08:52.826422 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826364    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c11b683-438e-445f-91f5-733cc6ff934a" containerID="26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb" exitCode=0
Sep 16 19:08:52.826422 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826416    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"
Sep 16 19:08:52.826855 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerDied","Data":"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"}
Sep 16 19:08:52.826855 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw" event={"ID":"5c11b683-438e-445f-91f5-733cc6ff934a","Type":"ContainerDied","Data":"39e83ae2a6170c69509ce98533838012523a533994277f2d8d6429c2c05d1633"}
Sep 16 19:08:52.826855 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826589    2563 scope.go:117] "RemoveContainer" containerID="3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"
Sep 16 19:08:52.827012 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.826982    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:08:52.834467 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.834450    2563 scope.go:117] "RemoveContainer" containerID="26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"
Sep 16 19:08:52.841303 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.841280    2563 scope.go:117] "RemoveContainer" containerID="3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"
Sep 16 19:08:52.841546 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:08:52.841529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0\": container with ID starting with 3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0 not found: ID does not exist" containerID="3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"
Sep 16 19:08:52.841613 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.841552    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0"} err="failed to get container status \"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0\": rpc error: code = NotFound desc = could not find container \"3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0\": container with ID starting with 3958ba2b7c5a4e08d977f07dd8528b5e56e27ffef4b7465f168be1f0bd16f7b0 not found: ID does not exist"
Sep 16 19:08:52.841613 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.841567    2563 scope.go:117] "RemoveContainer" containerID="26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"
Sep 16 19:08:52.841843 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:08:52.841805    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb\": container with ID starting with 26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb not found: ID does not exist" containerID="26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"
Sep 16 19:08:52.841908 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.841852    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb"} err="failed to get container status \"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb\": rpc error: code = NotFound desc = could not find container \"26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb\": container with ID starting with 26b87bb625c5fe47381e483f4d057fa8cfaa152f9e661c7d03aba3c10fddd8cb not found: ID does not exist"
Sep 16 19:08:52.923910 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.923887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls\") pod \"5c11b683-438e-445f-91f5-733cc6ff934a\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") "
Sep 16 19:08:52.924010 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.923914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config\") pod \"5c11b683-438e-445f-91f5-733cc6ff934a\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") "
Sep 16 19:08:52.924010 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.923989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzg4g\" (UniqueName: \"kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g\") pod \"5c11b683-438e-445f-91f5-733cc6ff934a\" (UID: \"5c11b683-438e-445f-91f5-733cc6ff934a\") "
Sep 16 19:08:52.924342 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.924318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c11b683-438e-445f-91f5-733cc6ff934a" (UID: "5c11b683-438e-445f-91f5-733cc6ff934a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:52.925828 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.925795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c11b683-438e-445f-91f5-733cc6ff934a" (UID: "5c11b683-438e-445f-91f5-733cc6ff934a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:52.925991 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:52.925975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g" (OuterVolumeSpecName: "kube-api-access-pzg4g") pod "5c11b683-438e-445f-91f5-733cc6ff934a" (UID: "5c11b683-438e-445f-91f5-733cc6ff934a"). InnerVolumeSpecName "kube-api-access-pzg4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:53.024953 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.024921    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c11b683-438e-445f-91f5-733cc6ff934a-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.024953 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.024952    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c11b683-438e-445f-91f5-733cc6ff934a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.024953 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.024963    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzg4g\" (UniqueName: \"kubernetes.io/projected/5c11b683-438e-445f-91f5-733cc6ff934a-kube-api-access-pzg4g\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:08:53.149073 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.149047    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 19:08:53.154445 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.154424    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a895e-predictor-79cfc466b9-vkzsw"]
Sep 16 19:08:53.722221 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:53.722188    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" path="/var/lib/kubelet/pods/5c11b683-438e-445f-91f5-733cc6ff934a/volumes"
Sep 16 19:08:56.826765 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:56.826738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:08:56.998055 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:56.998019    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:08:57.830956 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:57.830927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:08:57.831393 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:08:57.831366    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:09:00.297653 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.297621    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:09:00.298039 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.297830    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" containerID="cri-o://ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b" gracePeriod=30
Sep 16 19:09:00.357319 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.357287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.444020 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.443992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:09:00.444232 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.444211    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container" containerID="cri-o://1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f" gracePeriod=30
Sep 16 19:09:00.444286 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.444269    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kube-rbac-proxy" containerID="cri-o://2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955" gracePeriod=30
Sep 16 19:09:00.455085 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.455063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.581379 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581350    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:09:00.581628 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581616    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kube-rbac-proxy"
Sep 16 19:09:00.581670 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581630    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kube-rbac-proxy"
Sep 16 19:09:00.581670 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container"
Sep 16 19:09:00.581670 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581656    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container"
Sep 16 19:09:00.581756 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kube-rbac-proxy"
Sep 16 19:09:00.581756 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.581707    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c11b683-438e-445f-91f5-733cc6ff934a" containerName="kserve-container"
Sep 16 19:09:00.584471 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.584455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.588717 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.588697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-27911-predictor-serving-cert\""
Sep 16 19:09:00.589385 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.589364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-27911-kube-rbac-proxy-sar-config\""
Sep 16 19:09:00.596599 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.596578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.610459 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.610437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:09:00.679449 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.679425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxr7t\" (UniqueName: \"kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.679596 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.679460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.679596 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.679488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.774058 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.774017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.780074 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.780054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxr7t\" (UniqueName: \"kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.780134 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.780089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.780134 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.780111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.780715 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.780694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.782370 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.782350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.797713 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.797692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxr7t\" (UniqueName: \"kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t\") pod \"success-200-isvc-27911-predictor-8554d7997-sc9n6\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") " pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:00.858294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.858225    2563 generic.go:356] "Generic (PLEG): container finished" podID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerID="2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955" exitCode=2
Sep 16 19:09:00.858294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.858262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerDied","Data":"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"}
Sep 16 19:09:00.885333 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.885313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.894292 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:00.894275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:01.169688 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.169661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:09:01.172093 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:09:01.172066    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f64ec37_211a_4ebe_9a29_f20a249332d0.slice/crio-0b9d0c06877a1bf56c6eeb954e7fdb5f5c9aef93efdc46d8156c775140c56f93 WatchSource:0}: Error finding container 0b9d0c06877a1bf56c6eeb954e7fdb5f5c9aef93efdc46d8156c775140c56f93: Status 404 returned error can't find the container with id 0b9d0c06877a1bf56c6eeb954e7fdb5f5c9aef93efdc46d8156c775140c56f93
Sep 16 19:09:01.825252 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.825209    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:01.863096 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.863055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerStarted","Data":"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"}
Sep 16 19:09:01.863267 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.863125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:01.863267 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.863153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerStarted","Data":"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"}
Sep 16 19:09:01.863267 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.863180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:01.863267 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.863206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerStarted","Data":"0b9d0c06877a1bf56c6eeb954e7fdb5f5c9aef93efdc46d8156c775140c56f93"}
Sep 16 19:09:01.864505 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.864482    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:01.885071 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.885027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podStartSLOduration=1.885013128 podStartE2EDuration="1.885013128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:01.884657946 +0000 UTC m=+1430.623233434" watchObservedRunningTime="2026-09-16 19:09:01.885013128 +0000 UTC m=+1430.623588626"
Sep 16 19:09:01.997416 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.997384    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:01.997576 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:01.997474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:09:02.867353 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:02.867311    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:03.488616 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.488594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:09:03.600242 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.600187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") pod \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") "
Sep 16 19:09:03.600242 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.600236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config\") pod \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") "
Sep 16 19:09:03.600393 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.600317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msvmg\" (UniqueName: \"kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg\") pod \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\" (UID: \"8100d59c-40c8-4523-8dd5-e31a6c5cb54b\") "
Sep 16 19:09:03.600615 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.600576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8100d59c-40c8-4523-8dd5-e31a6c5cb54b" (UID: "8100d59c-40c8-4523-8dd5-e31a6c5cb54b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:03.602274 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.602250    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8100d59c-40c8-4523-8dd5-e31a6c5cb54b" (UID: "8100d59c-40c8-4523-8dd5-e31a6c5cb54b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:03.602369 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.602289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg" (OuterVolumeSpecName: "kube-api-access-msvmg") pod "8100d59c-40c8-4523-8dd5-e31a6c5cb54b" (UID: "8100d59c-40c8-4523-8dd5-e31a6c5cb54b"). InnerVolumeSpecName "kube-api-access-msvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:03.700915 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.700893    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvmg\" (UniqueName: \"kubernetes.io/projected/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-api-access-msvmg\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:03.701020 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.700917    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:03.701020 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.700932    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8100d59c-40c8-4523-8dd5-e31a6c5cb54b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:03.872626 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.872596    2563 generic.go:356] "Generic (PLEG): container finished" podID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerID="1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f" exitCode=0
Sep 16 19:09:03.873031 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.872657    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"
Sep 16 19:09:03.873031 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.872683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerDied","Data":"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"}
Sep 16 19:09:03.873031 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.872739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66" event={"ID":"8100d59c-40c8-4523-8dd5-e31a6c5cb54b","Type":"ContainerDied","Data":"c073e8d60f74eb4bf24e44162e7c6529f5567d6c9728b77cf1757a2c6e511ff1"}
Sep 16 19:09:03.873031 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.872766    2563 scope.go:117] "RemoveContainer" containerID="2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"
Sep 16 19:09:03.880277 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.880258    2563 scope.go:117] "RemoveContainer" containerID="1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"
Sep 16 19:09:03.887152 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.887135    2563 scope.go:117] "RemoveContainer" containerID="2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"
Sep 16 19:09:03.887407 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:03.887389    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955\": container with ID starting with 2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955 not found: ID does not exist" containerID="2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"
Sep 16 19:09:03.887449 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.887413    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955"} err="failed to get container status \"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955\": rpc error: code = NotFound desc = could not find container \"2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955\": container with ID starting with 2014b11f9aafc30eb078795968d846ed559034f0ad0e313deffc92862d940955 not found: ID does not exist"
Sep 16 19:09:03.887449 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.887428    2563 scope.go:117] "RemoveContainer" containerID="1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"
Sep 16 19:09:03.887677 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:03.887658    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f\": container with ID starting with 1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f not found: ID does not exist" containerID="1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"
Sep 16 19:09:03.887722 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.887684    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f"} err="failed to get container status \"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f\": rpc error: code = NotFound desc = could not find container \"1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f\": container with ID starting with 1e3b52a34c2208fa2ffc79b50600c17cc0340f36e2298f857c5879f2215d4e7f not found: ID does not exist"
Sep 16 19:09:03.892280 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.892261    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:09:03.903802 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:03.903783    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-443e6-predictor-c76bc877c-x6z66"]
Sep 16 19:09:05.722769 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:05.722732    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" path="/var/lib/kubelet/pods/8100d59c-40c8-4523-8dd5-e31a6c5cb54b/volumes"
Sep 16 19:09:06.825205 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:06.825173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:06.997343 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:06.997307    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:07.832372 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:07.832333    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:09:07.872315 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:07.872292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:07.872772 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:07.872749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:11.825617 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:11.825581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:11.826045 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:11.825689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:09:11.997875 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:11.997782    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:16.825910 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:16.825871    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:16.998077 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:16.998034    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:17.831749 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:17.831706    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:09:17.872808 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:17.872774    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:19.881769 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.881745    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:09:19.930264 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.930231    2563 generic.go:356] "Generic (PLEG): container finished" podID="da3911f4-2ade-4427-92f0-7f041c973e61" containerID="411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114" exitCode=0
Sep 16 19:09:19.930359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.930274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"
Sep 16 19:09:19.930359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.930285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" event={"ID":"da3911f4-2ade-4427-92f0-7f041c973e61","Type":"ContainerDied","Data":"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"}
Sep 16 19:09:19.930359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.930332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5" event={"ID":"da3911f4-2ade-4427-92f0-7f041c973e61","Type":"ContainerDied","Data":"f97d80de42720642502dd2d18085685e430f183de02bf10062216e9639854e43"}
Sep 16 19:09:19.930359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.930358    2563 scope.go:117] "RemoveContainer" containerID="411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"
Sep 16 19:09:19.937634 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.937620    2563 scope.go:117] "RemoveContainer" containerID="411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"
Sep 16 19:09:19.937889 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:19.937868    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114\": container with ID starting with 411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114 not found: ID does not exist" containerID="411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"
Sep 16 19:09:19.937985 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:19.937895    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114"} err="failed to get container status \"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114\": rpc error: code = NotFound desc = could not find container \"411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114\": container with ID starting with 411d5440ef0cd38e03514b1474a98e3a7d07ee28867260325ac8b048d281d114 not found: ID does not exist"
Sep 16 19:09:20.016709 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.016657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls\") pod \"da3911f4-2ade-4427-92f0-7f041c973e61\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") "
Sep 16 19:09:20.016709 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.016689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle\") pod \"da3911f4-2ade-4427-92f0-7f041c973e61\" (UID: \"da3911f4-2ade-4427-92f0-7f041c973e61\") "
Sep 16 19:09:20.017105 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.017078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "da3911f4-2ade-4427-92f0-7f041c973e61" (UID: "da3911f4-2ade-4427-92f0-7f041c973e61"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:20.018645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.018624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da3911f4-2ade-4427-92f0-7f041c973e61" (UID: "da3911f4-2ade-4427-92f0-7f041c973e61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:20.117411 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.117384    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da3911f4-2ade-4427-92f0-7f041c973e61-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.117411 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.117410    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da3911f4-2ade-4427-92f0-7f041c973e61-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.252998 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.252972    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:09:20.257151 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:20.257128    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a895e-569fdff559-p2sc5"]
Sep 16 19:09:21.728067 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:21.728030    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" path="/var/lib/kubelet/pods/da3911f4-2ade-4427-92f0-7f041c973e61/volumes"
Sep 16 19:09:21.825405 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:21.825371    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:26.825810 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:26.825763    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:09:27.831604 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:27.831561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 16 19:09:27.873589 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:27.873550    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:30.477881 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.477858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:09:30.494151 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.494131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls\") pod \"d5946f17-a43a-473c-b210-ce01616020bb\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") "
Sep 16 19:09:30.494266 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.494159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle\") pod \"d5946f17-a43a-473c-b210-ce01616020bb\" (UID: \"d5946f17-a43a-473c-b210-ce01616020bb\") "
Sep 16 19:09:30.494522 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.494500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d5946f17-a43a-473c-b210-ce01616020bb" (UID: "d5946f17-a43a-473c-b210-ce01616020bb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:30.496007 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.495983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5946f17-a43a-473c-b210-ce01616020bb" (UID: "d5946f17-a43a-473c-b210-ce01616020bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:30.595359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.595310    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5946f17-a43a-473c-b210-ce01616020bb-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.595359 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.595331    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5946f17-a43a-473c-b210-ce01616020bb-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:09:30.973481 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.973449    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5946f17-a43a-473c-b210-ce01616020bb" containerID="ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b" exitCode=137
Sep 16 19:09:30.973655 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.973497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"
Sep 16 19:09:30.973655 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.973539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" event={"ID":"d5946f17-a43a-473c-b210-ce01616020bb","Type":"ContainerDied","Data":"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"}
Sep 16 19:09:30.973655 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.973591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw" event={"ID":"d5946f17-a43a-473c-b210-ce01616020bb","Type":"ContainerDied","Data":"be54de767f77fe57c41d75a401365e75129f16fff0d4be1e49c4c21cfc963867"}
Sep 16 19:09:30.973655 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.973615    2563 scope.go:117] "RemoveContainer" containerID="ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"
Sep 16 19:09:30.982176 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.982156    2563 scope.go:117] "RemoveContainer" containerID="ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"
Sep 16 19:09:30.982442 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:30.982424    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b\": container with ID starting with ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b not found: ID does not exist" containerID="ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"
Sep 16 19:09:30.982505 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.982455    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b"} err="failed to get container status \"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b\": rpc error: code = NotFound desc = could not find container \"ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b\": container with ID starting with ce7a3eee3e24a1a769abce39f368f2da5a7b74e9bc327b8baf236b8f62611b2b not found: ID does not exist"
Sep 16 19:09:30.995342 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.995317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:09:30.999302 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:30.999282    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-443e6-7c548ddd5f-b4hgw"]
Sep 16 19:09:31.722609 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:31.722571    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5946f17-a43a-473c-b210-ce01616020bb" path="/var/lib/kubelet/pods/d5946f17-a43a-473c-b210-ce01616020bb/volumes"
Sep 16 19:09:36.117429 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:36.117397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:37.832750 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:37.832720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:09:37.873353 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:37.873318    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:09:47.873003 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:47.872968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:09:49.949662 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949900    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kube-rbac-proxy"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kube-rbac-proxy"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949939    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949946    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.949965    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.950017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3911f4-2ade-4427-92f0-7f041c973e61" containerName="sequence-graph-a895e"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.950024    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kube-rbac-proxy"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.950030    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5946f17-a43a-473c-b210-ce01616020bb" containerName="ensemble-graph-443e6"
Sep 16 19:09:49.950075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.950037    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8100d59c-40c8-4523-8dd5-e31a6c5cb54b" containerName="kserve-container"
Sep 16 19:09:49.952744 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.952726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:49.955539 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.955512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9450b-kube-rbac-proxy-sar-config\""
Sep 16 19:09:49.955539 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.955515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9450b-serving-cert\""
Sep 16 19:09:49.963080 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:49.963054    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:09:50.006331 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.006308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:50.028801 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.028783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.028914 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.028832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.129518 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.129493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.129607 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.129546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.129650 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:50.129626    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-9450b-serving-cert: secret "sequence-graph-9450b-serving-cert" not found
Sep 16 19:09:50.129690 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:09:50.129681    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls podName:9139b83b-964b-4bf2-be86-03fc1ee74e7e nodeName:}" failed. No retries permitted until 2026-09-16 19:09:50.62966628 +0000 UTC m=+1479.368241757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls") pod "sequence-graph-9450b-587fc5878c-n2qk6" (UID: "9139b83b-964b-4bf2-be86-03fc1ee74e7e") : secret "sequence-graph-9450b-serving-cert" not found
Sep 16 19:09:50.130103 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.130086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.506075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.506041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:50.632338 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.632312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.634709 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.634688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") pod \"sequence-graph-9450b-587fc5878c-n2qk6\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") " pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:50.863413 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:50.863389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:51.140181 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:51.140057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:09:51.142999 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:09:51.142965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9139b83b_964b_4bf2_be86_03fc1ee74e7e.slice/crio-141ef28b2496a7d118a6c9cc71219e78edc7e2aa51f7ce3848a8f68619537560 WatchSource:0}: Error finding container 141ef28b2496a7d118a6c9cc71219e78edc7e2aa51f7ce3848a8f68619537560: Status 404 returned error can't find the container with id 141ef28b2496a7d118a6c9cc71219e78edc7e2aa51f7ce3848a8f68619537560
Sep 16 19:09:52.056554 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:52.056519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" event={"ID":"9139b83b-964b-4bf2-be86-03fc1ee74e7e","Type":"ContainerStarted","Data":"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"}
Sep 16 19:09:52.056716 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:52.056594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" event={"ID":"9139b83b-964b-4bf2-be86-03fc1ee74e7e","Type":"ContainerStarted","Data":"141ef28b2496a7d118a6c9cc71219e78edc7e2aa51f7ce3848a8f68619537560"}
Sep 16 19:09:52.056716 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:52.056623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:58.065659 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:58.065621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:09:58.084602 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:09:58.084553    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podStartSLOduration=9.084537156 podStartE2EDuration="9.084537156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:52.073999828 +0000 UTC m=+1480.812575327" watchObservedRunningTime="2026-09-16 19:09:58.084537156 +0000 UTC m=+1486.823112657"
Sep 16 19:10:00.043027 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.042957    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:10:00.043379 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.043156    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" containerID="cri-o://82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7" gracePeriod=30
Sep 16 19:10:00.071409 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.071385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.110718 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.110693    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:10:00.110949 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.110907    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container" containerID="cri-o://6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d" gracePeriod=30
Sep 16 19:10:00.111083 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.110938    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kube-rbac-proxy" containerID="cri-o://d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422" gracePeriod=30
Sep 16 19:10:00.170494 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.170466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.256548 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.256518    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:10:00.259700 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.259684    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.262801 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.262778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53a10-predictor-serving-cert\""
Sep 16 19:10:00.262991 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.262866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53a10-kube-rbac-proxy-sar-config\""
Sep 16 19:10:00.271062 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.271042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:10:00.280636 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.280614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.298994 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.298942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.298994 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.298971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.299112 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.299005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4njp\" (UniqueName: \"kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.399367 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.399344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4njp\" (UniqueName: \"kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.399475 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.399386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.399475 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.399406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.399581 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:00.399496    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-53a10-predictor-serving-cert: secret "success-200-isvc-53a10-predictor-serving-cert" not found
Sep 16 19:10:00.399581 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:00.399553    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls podName:9b66c1c9-85af-4ec4-a37b-9ac29daca655 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:00.899538623 +0000 UTC m=+1489.638114100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls") pod "success-200-isvc-53a10-predictor-86b75c99c-66zjg" (UID: "9b66c1c9-85af-4ec4-a37b-9ac29daca655") : secret "success-200-isvc-53a10-predictor-serving-cert" not found
Sep 16 19:10:00.399998 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.399981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.418283 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.418257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.418577 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.418557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4njp\" (UniqueName: \"kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.501010 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.500982    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:10:00.504276 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.504262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.507387 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.507367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-27911-serving-cert\""
Sep 16 19:10:00.507635 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.507618    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-27911-kube-rbac-proxy-sar-config\""
Sep 16 19:10:00.515307 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.515288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:10:00.581222 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.581161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.600371 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.600349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.600433 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.600399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.684185 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.684162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.700955 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.700932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.701017 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.700974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.701077 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:00.701062    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-27911-serving-cert: secret "ensemble-graph-27911-serving-cert" not found
Sep 16 19:10:00.701125 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:00.701114    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls podName:cc0f9c39-cd1a-4645-b44f-03fa6432d7c1 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:01.201101786 +0000 UTC m=+1489.939677263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls") pod "ensemble-graph-27911-77bb65458-2c7ns" (UID: "cc0f9c39-cd1a-4645-b44f-03fa6432d7c1") : secret "ensemble-graph-27911-serving-cert" not found
Sep 16 19:10:00.701517 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.701499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:00.902900 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.902874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.905227 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.905207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") pod \"success-200-isvc-53a10-predictor-86b75c99c-66zjg\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") " pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:00.982268 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:00.982238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:01.090000 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.089975    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerID="d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422" exitCode=2
Sep 16 19:10:01.090305 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.090019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerDied","Data":"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"}
Sep 16 19:10:01.169260 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.169201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:01.205531 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.205493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:01.208095 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.208068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") pod \"ensemble-graph-27911-77bb65458-2c7ns\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") " pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:01.414181 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.414152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:01.440515 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.440468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:10:01.443066 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:10:01.443038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b66c1c9_85af_4ec4_a37b_9ac29daca655.slice/crio-a7bd06f4b4ba1aaee6d4cd14bbd55e16b5f97f84f260282084b1c25b2fb18613 WatchSource:0}: Error finding container a7bd06f4b4ba1aaee6d4cd14bbd55e16b5f97f84f260282084b1c25b2fb18613: Status 404 returned error can't find the container with id a7bd06f4b4ba1aaee6d4cd14bbd55e16b5f97f84f260282084b1c25b2fb18613
Sep 16 19:10:01.687531 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:01.687376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:10:01.689433 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:10:01.689407    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0f9c39_cd1a_4645_b44f_03fa6432d7c1.slice/crio-636c996ee60432730431462b65caecfa6b1692fbaeaf0a9351a1c148824372db WatchSource:0}: Error finding container 636c996ee60432730431462b65caecfa6b1692fbaeaf0a9351a1c148824372db: Status 404 returned error can't find the container with id 636c996ee60432730431462b65caecfa6b1692fbaeaf0a9351a1c148824372db
Sep 16 19:10:02.094742 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.094653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" event={"ID":"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1","Type":"ContainerStarted","Data":"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"}
Sep 16 19:10:02.095169 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.094748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" event={"ID":"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1","Type":"ContainerStarted","Data":"636c996ee60432730431462b65caecfa6b1692fbaeaf0a9351a1c148824372db"}
Sep 16 19:10:02.095169 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.094780    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:02.096640 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.096614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerStarted","Data":"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"}
Sep 16 19:10:02.096765 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.096661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerStarted","Data":"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"}
Sep 16 19:10:02.096765 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.096684    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:02.096765 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.096703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerStarted","Data":"a7bd06f4b4ba1aaee6d4cd14bbd55e16b5f97f84f260282084b1c25b2fb18613"}
Sep 16 19:10:02.096765 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.096721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:02.098016 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.097992    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:02.113242 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.113203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podStartSLOduration=2.113190688 podStartE2EDuration="2.113190688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:02.112555386 +0000 UTC m=+1490.851130883" watchObservedRunningTime="2026-09-16 19:10:02.113190688 +0000 UTC m=+1490.851766213"
Sep 16 19:10:02.142931 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.142889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podStartSLOduration=2.142880115 podStartE2EDuration="2.142880115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:02.141683744 +0000 UTC m=+1490.880259241" watchObservedRunningTime="2026-09-16 19:10:02.142880115 +0000 UTC m=+1490.881455613"
Sep 16 19:10:02.855792 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.855771    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:10:02.919394 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.919328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") pod \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") "
Sep 16 19:10:02.919394 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.919365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdzv2\" (UniqueName: \"kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2\") pod \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") "
Sep 16 19:10:02.919574 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.919485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config\") pod \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\" (UID: \"2e236bb2-a567-40ef-8a2b-dcaea42291c3\") "
Sep 16 19:10:02.919856 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.919834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e236bb2-a567-40ef-8a2b-dcaea42291c3" (UID: "2e236bb2-a567-40ef-8a2b-dcaea42291c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:02.921387 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.921364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2" (OuterVolumeSpecName: "kube-api-access-hdzv2") pod "2e236bb2-a567-40ef-8a2b-dcaea42291c3" (UID: "2e236bb2-a567-40ef-8a2b-dcaea42291c3"). InnerVolumeSpecName "kube-api-access-hdzv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:02.921475 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:02.921368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e236bb2-a567-40ef-8a2b-dcaea42291c3" (UID: "2e236bb2-a567-40ef-8a2b-dcaea42291c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:03.020161 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.020128    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e236bb2-a567-40ef-8a2b-dcaea42291c3-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:10:03.020161 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.020160    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzv2\" (UniqueName: \"kubernetes.io/projected/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-api-access-hdzv2\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:10:03.020161 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.020171    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e236bb2-a567-40ef-8a2b-dcaea42291c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:10:03.063954 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.063925    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:03.101895 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.101875    2563 generic.go:356] "Generic (PLEG): container finished" podID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerID="6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d" exitCode=0
Sep 16 19:10:03.102199 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.101927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"
Sep 16 19:10:03.102199 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.101962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerDied","Data":"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"}
Sep 16 19:10:03.102199 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.102019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" event={"ID":"2e236bb2-a567-40ef-8a2b-dcaea42291c3","Type":"ContainerDied","Data":"31c769b84262667d41ae15483d02c6b0f04d26c6b94e5b9ddd28d8a717d97ebd"}
Sep 16 19:10:03.102199 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.102048    2563 scope.go:117] "RemoveContainer" containerID="d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"
Sep 16 19:10:03.102585 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.102563    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:03.110214 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.110197    2563 scope.go:117] "RemoveContainer" containerID="6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"
Sep 16 19:10:03.117111 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.117098    2563 scope.go:117] "RemoveContainer" containerID="d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"
Sep 16 19:10:03.117350 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:03.117332    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422\": container with ID starting with d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422 not found: ID does not exist" containerID="d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"
Sep 16 19:10:03.117393 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.117357    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422"} err="failed to get container status \"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422\": rpc error: code = NotFound desc = could not find container \"d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422\": container with ID starting with d188cdb845ffec954ac9ad2bc43970c03359efea96cf3abff15f2c1cd6276422 not found: ID does not exist"
Sep 16 19:10:03.117393 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.117370    2563 scope.go:117] "RemoveContainer" containerID="6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"
Sep 16 19:10:03.117585 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:03.117572    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d\": container with ID starting with 6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d not found: ID does not exist" containerID="6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"
Sep 16 19:10:03.117651 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.117589    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d"} err="failed to get container status \"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d\": rpc error: code = NotFound desc = could not find container \"6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d\": container with ID starting with 6bb4fa7c98581fc00fd4eeed3862ce208e880a166b02383f8a49dfec8a4e938d not found: ID does not exist"
Sep 16 19:10:03.124065 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.124046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:10:03.128419 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.128401    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8"]
Sep 16 19:10:03.723413 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.723381    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" path="/var/lib/kubelet/pods/2e236bb2-a567-40ef-8a2b-dcaea42291c3/volumes"
Sep 16 19:10:03.828026 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:03.827992    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9450b-predictor-67ccb68c65-x62p8" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:10:08.064706 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:08.064666    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:08.107039 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:08.107001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:10:08.107738 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:08.107716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:10:08.108193 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:08.108166    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:13.064429 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:13.064380    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:13.064810 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:13.064508    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:10:14.754242 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:14.754207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:18.064211 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:18.064172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:18.108635 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:18.108600    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:23.064048 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:23.064010    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:28.064185 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:28.064145    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:10:28.109095 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:28.109060    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:30.176808 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.176784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:10:30.205345 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.205318    2563 generic.go:356] "Generic (PLEG): container finished" podID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerID="82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7" exitCode=0
Sep 16 19:10:30.205434 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.205349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"
Sep 16 19:10:30.205434 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.205362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" event={"ID":"9139b83b-964b-4bf2-be86-03fc1ee74e7e","Type":"ContainerDied","Data":"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"}
Sep 16 19:10:30.205434 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.205408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6" event={"ID":"9139b83b-964b-4bf2-be86-03fc1ee74e7e","Type":"ContainerDied","Data":"141ef28b2496a7d118a6c9cc71219e78edc7e2aa51f7ce3848a8f68619537560"}
Sep 16 19:10:30.205561 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.205438    2563 scope.go:117] "RemoveContainer" containerID="82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"
Sep 16 19:10:30.212731 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.212714    2563 scope.go:117] "RemoveContainer" containerID="82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"
Sep 16 19:10:30.213034 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:10:30.213014    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7\": container with ID starting with 82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7 not found: ID does not exist" containerID="82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"
Sep 16 19:10:30.213099 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.213042    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7"} err="failed to get container status \"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7\": rpc error: code = NotFound desc = could not find container \"82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7\": container with ID starting with 82c9d54dd0afc1305938c32b73a2210ea24b9c9cd7b330a19b226ee168b31ee7 not found: ID does not exist"
Sep 16 19:10:30.306162 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.306112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle\") pod \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") "
Sep 16 19:10:30.306240 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.306168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") pod \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\" (UID: \"9139b83b-964b-4bf2-be86-03fc1ee74e7e\") "
Sep 16 19:10:30.306502 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.306477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9139b83b-964b-4bf2-be86-03fc1ee74e7e" (UID: "9139b83b-964b-4bf2-be86-03fc1ee74e7e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:30.308135 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.308118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9139b83b-964b-4bf2-be86-03fc1ee74e7e" (UID: "9139b83b-964b-4bf2-be86-03fc1ee74e7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:30.407184 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.407162    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9139b83b-964b-4bf2-be86-03fc1ee74e7e-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:10:30.407184 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.407181    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9139b83b-964b-4bf2-be86-03fc1ee74e7e-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:10:30.531839 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.531802    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:10:30.537477 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:30.537455    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9450b-587fc5878c-n2qk6"]
Sep 16 19:10:31.722904 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:31.722871    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" path="/var/lib/kubelet/pods/9139b83b-964b-4bf2-be86-03fc1ee74e7e/volumes"
Sep 16 19:10:38.108929 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:38.108837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 19:10:48.109464 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:10:48.109437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:11:00.303114 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303079    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303470    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kube-rbac-proxy"
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303489    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kube-rbac-proxy"
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303515    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b"
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303523    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b"
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303536    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container"
Sep 16 19:11:00.303556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303543    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container"
Sep 16 19:11:00.303862 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9139b83b-964b-4bf2-be86-03fc1ee74e7e" containerName="sequence-graph-9450b"
Sep 16 19:11:00.303862 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kserve-container"
Sep 16 19:11:00.303862 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.303636    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e236bb2-a567-40ef-8a2b-dcaea42291c3" containerName="kube-rbac-proxy"
Sep 16 19:11:00.306618 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.306596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.309598 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.309579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53a10-serving-cert\""
Sep 16 19:11:00.309747 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.309725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53a10-kube-rbac-proxy-sar-config\""
Sep 16 19:11:00.315343 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.315314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.315704 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.315683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:11:00.317878 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.317857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.317969 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.317916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.418751 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.418729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.418864 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.418767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.418911 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:11:00.418894    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-53a10-serving-cert: secret "sequence-graph-53a10-serving-cert" not found
Sep 16 19:11:00.418976 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:11:00.418965    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls podName:d6c2b4d2-499b-4975-901d-7a6f28e71ab0 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:00.918947745 +0000 UTC m=+1549.657523235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls") pod "sequence-graph-53a10-756f44fbbb-hpvpp" (UID: "d6c2b4d2-499b-4975-901d-7a6f28e71ab0") : secret "sequence-graph-53a10-serving-cert" not found
Sep 16 19:11:00.419415 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.419398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.922240 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:00.922208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:00.922360 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:11:00.922320    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-53a10-serving-cert: secret "sequence-graph-53a10-serving-cert" not found
Sep 16 19:11:00.922402 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:11:00.922368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls podName:d6c2b4d2-499b-4975-901d-7a6f28e71ab0 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:01.922356032 +0000 UTC m=+1550.660931507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls") pod "sequence-graph-53a10-756f44fbbb-hpvpp" (UID: "d6c2b4d2-499b-4975-901d-7a6f28e71ab0") : secret "sequence-graph-53a10-serving-cert" not found
Sep 16 19:11:01.036254 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:01.036228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:01.928032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:01.927997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:01.930460 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:01.930440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"sequence-graph-53a10-756f44fbbb-hpvpp\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") " pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:02.117289 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:02.117262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:02.388412 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:02.388313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:11:02.390581 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:11:02.390554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c2b4d2_499b_4975_901d_7a6f28e71ab0.slice/crio-9d2d206d40d4e146518d7054e431e0be67e88b062f88c3a5a91133b008058790 WatchSource:0}: Error finding container 9d2d206d40d4e146518d7054e431e0be67e88b062f88c3a5a91133b008058790: Status 404 returned error can't find the container with id 9d2d206d40d4e146518d7054e431e0be67e88b062f88c3a5a91133b008058790
Sep 16 19:11:03.328904 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:03.328864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" event={"ID":"d6c2b4d2-499b-4975-901d-7a6f28e71ab0","Type":"ContainerStarted","Data":"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"}
Sep 16 19:11:03.329362 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:03.328926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" event={"ID":"d6c2b4d2-499b-4975-901d-7a6f28e71ab0","Type":"ContainerStarted","Data":"9d2d206d40d4e146518d7054e431e0be67e88b062f88c3a5a91133b008058790"}
Sep 16 19:11:03.329362 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:03.329001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:11:03.348685 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:03.348635    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podStartSLOduration=3.348622439 podStartE2EDuration="3.348622439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:03.347132337 +0000 UTC m=+1552.085707835" watchObservedRunningTime="2026-09-16 19:11:03.348622439 +0000 UTC m=+1552.087197938"
Sep 16 19:11:09.338298 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:11:09.338267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:13:11.731782 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:13:11.731752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:59.252754 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:13:59.252713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:59.328858 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:13:59.328807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:15.195063 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.194989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:18:15.197590 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.195193    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" containerID="cri-o://4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5" gracePeriod=30
Sep 16 19:18:15.260298 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.260261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:15.351179 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.351150    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:18:15.351412 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.351389    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" containerID="cri-o://6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564" gracePeriod=30
Sep 16 19:18:15.351491 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.351394    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kube-rbac-proxy" containerID="cri-o://67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561" gracePeriod=30
Sep 16 19:18:15.470851 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.470796    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:18:15.474755 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.474734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.477612 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.477583    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3bba6-kube-rbac-proxy-sar-config\""
Sep 16 19:18:15.477717 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.477648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3bba6-predictor-serving-cert\""
Sep 16 19:18:15.488169 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.488149    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:18:15.598593 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.598568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.598746 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.598615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.598746 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.598684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vfb\" (UniqueName: \"kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.699006 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.698978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.699145 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.699022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vfb\" (UniqueName: \"kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.699145 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.699061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.699636 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.699614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.701411 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.701390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.712449 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.712425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vfb\" (UniqueName: \"kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb\") pod \"success-200-isvc-3bba6-predictor-65498c7465-jfhbj\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") " pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.785590 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.785536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:15.919003 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.918974    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerID="67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561" exitCode=2
Sep 16 19:18:15.919131 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:15.919050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerDied","Data":"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"}
Sep 16 19:18:16.058242 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.058218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:18:16.060793 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:18:16.060763    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7380ca_84ba_43db_be17_cc87934737ac.slice/crio-1aa5800c648420038ca864972715e392ca17f7082ebb91eab08bb1e951ada80f WatchSource:0}: Error finding container 1aa5800c648420038ca864972715e392ca17f7082ebb91eab08bb1e951ada80f: Status 404 returned error can't find the container with id 1aa5800c648420038ca864972715e392ca17f7082ebb91eab08bb1e951ada80f
Sep 16 19:18:16.924644 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.924607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerStarted","Data":"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"}
Sep 16 19:18:16.925113 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.924669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerStarted","Data":"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"}
Sep 16 19:18:16.925113 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.924693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerStarted","Data":"1aa5800c648420038ca864972715e392ca17f7082ebb91eab08bb1e951ada80f"}
Sep 16 19:18:16.925113 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.924740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:16.946280 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.946215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podStartSLOduration=1.9461941010000001 podStartE2EDuration="1.946194101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:16.945624661 +0000 UTC m=+1985.684200159" watchObservedRunningTime="2026-09-16 19:18:16.946194101 +0000 UTC m=+1985.684769600"
Sep 16 19:18:16.999647 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:16.999619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:17.868188 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:17.868148    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 16 19:18:17.873519 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:17.873496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 16 19:18:17.928510 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:17.928477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:17.929644 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:17.929620    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:18:18.096466 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.096443    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:18:18.107539 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.107515    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:18.216783 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.216724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxr7t\" (UniqueName: \"kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t\") pod \"4f64ec37-211a-4ebe-9a29-f20a249332d0\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") "
Sep 16 19:18:18.216929 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.216789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls\") pod \"4f64ec37-211a-4ebe-9a29-f20a249332d0\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") "
Sep 16 19:18:18.216929 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.216835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config\") pod \"4f64ec37-211a-4ebe-9a29-f20a249332d0\" (UID: \"4f64ec37-211a-4ebe-9a29-f20a249332d0\") "
Sep 16 19:18:18.217203 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.217179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f64ec37-211a-4ebe-9a29-f20a249332d0" (UID: "4f64ec37-211a-4ebe-9a29-f20a249332d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:18.218676 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.218658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f64ec37-211a-4ebe-9a29-f20a249332d0" (UID: "4f64ec37-211a-4ebe-9a29-f20a249332d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:18.218878 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.218864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t" (OuterVolumeSpecName: "kube-api-access-hxr7t") pod "4f64ec37-211a-4ebe-9a29-f20a249332d0" (UID: "4f64ec37-211a-4ebe-9a29-f20a249332d0"). InnerVolumeSpecName "kube-api-access-hxr7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:18.317374 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.317350    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.317374 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.317370    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxr7t\" (UniqueName: \"kubernetes.io/projected/4f64ec37-211a-4ebe-9a29-f20a249332d0-kube-api-access-hxr7t\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.317482 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.317379    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f64ec37-211a-4ebe-9a29-f20a249332d0-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.933722 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.933692    2563 generic.go:356] "Generic (PLEG): container finished" podID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerID="6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564" exitCode=0
Sep 16 19:18:18.934148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.933739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"
Sep 16 19:18:18.934148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.933773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerDied","Data":"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"}
Sep 16 19:18:18.934148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.933840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6" event={"ID":"4f64ec37-211a-4ebe-9a29-f20a249332d0","Type":"ContainerDied","Data":"0b9d0c06877a1bf56c6eeb954e7fdb5f5c9aef93efdc46d8156c775140c56f93"}
Sep 16 19:18:18.934148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.933868    2563 scope.go:117] "RemoveContainer" containerID="67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"
Sep 16 19:18:18.934362 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.934208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:18:18.941758 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.941728    2563 scope.go:117] "RemoveContainer" containerID="6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"
Sep 16 19:18:18.948650 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.948631    2563 scope.go:117] "RemoveContainer" containerID="67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"
Sep 16 19:18:18.948919 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:18:18.948899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561\": container with ID starting with 67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561 not found: ID does not exist" containerID="67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"
Sep 16 19:18:18.949004 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.948932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561"} err="failed to get container status \"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561\": rpc error: code = NotFound desc = could not find container \"67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561\": container with ID starting with 67617746f6ab0f05eec85ab45aedbc30446267dd91232ee20a1aaf31aacf0561 not found: ID does not exist"
Sep 16 19:18:18.949004 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.948948    2563 scope.go:117] "RemoveContainer" containerID="6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"
Sep 16 19:18:18.949186 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:18:18.949171    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564\": container with ID starting with 6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564 not found: ID does not exist" containerID="6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"
Sep 16 19:18:18.949225 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.949189    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564"} err="failed to get container status \"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564\": rpc error: code = NotFound desc = could not find container \"6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564\": container with ID starting with 6aabe59b107f9ecc1cd09c43bb85a435a1d8774710d0d2ba60e2d502b726b564 not found: ID does not exist"
Sep 16 19:18:18.958890 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.958872    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:18:18.963173 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:18.963152    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-27911-predictor-8554d7997-sc9n6"]
Sep 16 19:18:19.722531 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:19.722499    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" path="/var/lib/kubelet/pods/4f64ec37-211a-4ebe-9a29-f20a249332d0/volumes"
Sep 16 19:18:23.105779 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:23.105728    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:23.939998 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:23.939961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:18:23.940560 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:23.940529    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:18:28.106262 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:28.106222    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:28.106634 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:28.106317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:18:33.106555 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:33.106518    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:33.940424 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:33.940386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:18:38.106249 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:38.106204    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:43.107164 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:43.107115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:18:43.941246 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:43.941208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:18:45.371804 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.371782    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:18:45.391886 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.391861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle\") pod \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") "
Sep 16 19:18:45.392003 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.391909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") pod \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\" (UID: \"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1\") "
Sep 16 19:18:45.392214 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.392191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" (UID: "cc0f9c39-cd1a-4645-b44f-03fa6432d7c1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:45.394003 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.393977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" (UID: "cc0f9c39-cd1a-4645-b44f-03fa6432d7c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:45.493105 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.493044    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:18:45.493105 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:45.493067    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:18:46.034219 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.034185    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerID="4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5" exitCode=0
Sep 16 19:18:46.034381 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.034222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"
Sep 16 19:18:46.034381 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.034248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" event={"ID":"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1","Type":"ContainerDied","Data":"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"}
Sep 16 19:18:46.034381 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.034288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns" event={"ID":"cc0f9c39-cd1a-4645-b44f-03fa6432d7c1","Type":"ContainerDied","Data":"636c996ee60432730431462b65caecfa6b1692fbaeaf0a9351a1c148824372db"}
Sep 16 19:18:46.034381 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.034308    2563 scope.go:117] "RemoveContainer" containerID="4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"
Sep 16 19:18:46.042422 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.042399    2563 scope.go:117] "RemoveContainer" containerID="4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"
Sep 16 19:18:46.042711 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:18:46.042694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5\": container with ID starting with 4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5 not found: ID does not exist" containerID="4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"
Sep 16 19:18:46.042780 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.042721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5"} err="failed to get container status \"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5\": rpc error: code = NotFound desc = could not find container \"4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5\": container with ID starting with 4888313a0336510174b487fd79c20597c61a63a4b11b0bee7b121f8c972a1ca5 not found: ID does not exist"
Sep 16 19:18:46.051623 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.051594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:18:46.054650 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:46.054631    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-27911-77bb65458-2c7ns"]
Sep 16 19:18:47.722657 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:47.722621    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" path="/var/lib/kubelet/pods/cc0f9c39-cd1a-4645-b44f-03fa6432d7c1/volumes"
Sep 16 19:18:53.940648 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:18:53.940604    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 19:19:03.942067 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:03.942025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:19:14.977976 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:14.977941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:19:14.978416 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:14.978149    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" containerID="cri-o://d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078" gracePeriod=30
Sep 16 19:19:15.049791 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.049757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.106635 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.106605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:19:15.106882 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.106852    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kube-rbac-proxy" containerID="cri-o://b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7" gracePeriod=30
Sep 16 19:19:15.106955 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.106919    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container" containerID="cri-o://83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421" gracePeriod=30
Sep 16 19:19:15.156101 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.156077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.249244 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249168    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:19:15.249557 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kube-rbac-proxy"
Sep 16 19:19:15.249645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kube-rbac-proxy"
Sep 16 19:19:15.249645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249593    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911"
Sep 16 19:19:15.249645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249602    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911"
Sep 16 19:19:15.249645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249612    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container"
Sep 16 19:19:15.249645 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container"
Sep 16 19:19:15.249914 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249685    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0f9c39-cd1a-4645-b44f-03fa6432d7c1" containerName="ensemble-graph-27911"
Sep 16 19:19:15.249914 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kube-rbac-proxy"
Sep 16 19:19:15.249914 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.249713    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f64ec37-211a-4ebe-9a29-f20a249332d0" containerName="kserve-container"
Sep 16 19:19:15.252654 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.252632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.255674 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.255652    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7f5a4-predictor-serving-cert\""
Sep 16 19:19:15.255796 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.255777    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7f5a4-kube-rbac-proxy-sar-config\""
Sep 16 19:19:15.256482 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.256459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.266262 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.266242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:19:15.289425 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.289401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.289502 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.289445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.289502 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.289461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjjp\" (UniqueName: \"kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.357454 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.357428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.389905 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.389878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.390026 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.389956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.390026 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.389981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjjp\" (UniqueName: \"kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.390205 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:15.390174    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-serving-cert: secret "success-200-isvc-7f5a4-predictor-serving-cert" not found
Sep 16 19:19:15.390269 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:15.390262    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls podName:743bd70b-3137-4d57-8c52-9c019a6aa05d nodeName:}" failed. No retries permitted until 2026-09-16 19:19:15.890241828 +0000 UTC m=+2044.628817324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls") pod "success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" (UID: "743bd70b-3137-4d57-8c52-9c019a6aa05d") : secret "success-200-isvc-7f5a4-predictor-serving-cert" not found
Sep 16 19:19:15.390694 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.390673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.402613 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.402582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjjp\" (UniqueName: \"kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.436082 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.436060    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:15.439225 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.439211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.442179 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.442159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-3bba6-serving-cert\""
Sep 16 19:19:15.442447 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.442425    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-3bba6-kube-rbac-proxy-sar-config\""
Sep 16 19:19:15.449612 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.449592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:15.451342 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.451322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.490689 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.490668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.490768 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.490716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.591147 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.591094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.591147 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.591143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.591277 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:15.591229    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-3bba6-serving-cert: secret "splitter-graph-3bba6-serving-cert" not found
Sep 16 19:19:15.591313 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:15.591279    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls podName:7154dd1e-7dd6-4a7d-b671-6ed9d1776058 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:16.091263119 +0000 UTC m=+2044.829838612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls") pod "splitter-graph-3bba6-879cd4d78-6rx4b" (UID: "7154dd1e-7dd6-4a7d-b671-6ed9d1776058") : secret "splitter-graph-3bba6-serving-cert" not found
Sep 16 19:19:15.591678 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.591659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:15.647570 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.647545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.746932 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.746908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.893195 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.893175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:15.895525 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:15.895508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") pod \"success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") " pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:16.068513 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.068480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:16.097163 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.097136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:16.099415 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.099394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") pod \"splitter-graph-3bba6-879cd4d78-6rx4b\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") " pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:16.148151 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.148081    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerID="b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7" exitCode=2
Sep 16 19:19:16.148151 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.148146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerDied","Data":"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"}
Sep 16 19:19:16.161950 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.161928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:16.349046 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.349015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:16.429554 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.429533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:19:16.431334 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:19:16.431306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743bd70b_3137_4d57_8c52_9c019a6aa05d.slice/crio-cdb4972e056c57548b2d2957ad5c4e007ae3e51ea838a98b28467c169d7bd9fd WatchSource:0}: Error finding container cdb4972e056c57548b2d2957ad5c4e007ae3e51ea838a98b28467c169d7bd9fd: Status 404 returned error can't find the container with id cdb4972e056c57548b2d2957ad5c4e007ae3e51ea838a98b28467c169d7bd9fd
Sep 16 19:19:16.618190 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:16.618169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:16.620060 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:19:16.620036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7154dd1e_7dd6_4a7d_b671_6ed9d1776058.slice/crio-e49baa1a115dfcdb7ef784c5369cfbff2c4bab66f83148e96f54b9c146be35d2 WatchSource:0}: Error finding container e49baa1a115dfcdb7ef784c5369cfbff2c4bab66f83148e96f54b9c146be35d2: Status 404 returned error can't find the container with id e49baa1a115dfcdb7ef784c5369cfbff2c4bab66f83148e96f54b9c146be35d2
Sep 16 19:19:17.153196 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.153162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" event={"ID":"7154dd1e-7dd6-4a7d-b671-6ed9d1776058","Type":"ContainerStarted","Data":"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"}
Sep 16 19:19:17.153596 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.153212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" event={"ID":"7154dd1e-7dd6-4a7d-b671-6ed9d1776058","Type":"ContainerStarted","Data":"e49baa1a115dfcdb7ef784c5369cfbff2c4bab66f83148e96f54b9c146be35d2"}
Sep 16 19:19:17.153596 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.153249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:17.155005 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.154981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerStarted","Data":"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"}
Sep 16 19:19:17.155115 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.155032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerStarted","Data":"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"}
Sep 16 19:19:17.155115 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.155058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:17.155115 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.155077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerStarted","Data":"cdb4972e056c57548b2d2957ad5c4e007ae3e51ea838a98b28467c169d7bd9fd"}
Sep 16 19:19:17.155115 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.155105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:17.156573 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.156549    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:17.177250 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:17.177204    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podStartSLOduration=2.177186415 podStartE2EDuration="2.177186415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:17.176377075 +0000 UTC m=+2045.914952574" watchObservedRunningTime="2026-09-16 19:19:17.177186415 +0000 UTC m=+2045.915761913"
Sep 16 19:19:18.048204 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.048176    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:19:18.070586 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.070542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podStartSLOduration=3.070527678 podStartE2EDuration="3.070527678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:17.201532876 +0000 UTC m=+2045.940108371" watchObservedRunningTime="2026-09-16 19:19:18.070527678 +0000 UTC m=+2046.809103175"
Sep 16 19:19:18.111683 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.111632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4njp\" (UniqueName: \"kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp\") pod \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") "
Sep 16 19:19:18.111782 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.111688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config\") pod \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") "
Sep 16 19:19:18.111782 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.111716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") pod \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\" (UID: \"9b66c1c9-85af-4ec4-a37b-9ac29daca655\") "
Sep 16 19:19:18.112086 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.112059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b66c1c9-85af-4ec4-a37b-9ac29daca655" (UID: "9b66c1c9-85af-4ec4-a37b-9ac29daca655"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:18.113632 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.113608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b66c1c9-85af-4ec4-a37b-9ac29daca655" (UID: "9b66c1c9-85af-4ec4-a37b-9ac29daca655"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:18.113632 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.113609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp" (OuterVolumeSpecName: "kube-api-access-f4njp") pod "9b66c1c9-85af-4ec4-a37b-9ac29daca655" (UID: "9b66c1c9-85af-4ec4-a37b-9ac29daca655"). InnerVolumeSpecName "kube-api-access-f4njp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:18.159921 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.159897    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerID="83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421" exitCode=0
Sep 16 19:19:18.160200 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.159941    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"
Sep 16 19:19:18.160200 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.159982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerDied","Data":"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"}
Sep 16 19:19:18.160200 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.160035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg" event={"ID":"9b66c1c9-85af-4ec4-a37b-9ac29daca655","Type":"ContainerDied","Data":"a7bd06f4b4ba1aaee6d4cd14bbd55e16b5f97f84f260282084b1c25b2fb18613"}
Sep 16 19:19:18.160200 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.160061    2563 scope.go:117] "RemoveContainer" containerID="b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"
Sep 16 19:19:18.160448 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.160417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:18.168265 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.168249    2563 scope.go:117] "RemoveContainer" containerID="83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"
Sep 16 19:19:18.175791 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.175771    2563 scope.go:117] "RemoveContainer" containerID="b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"
Sep 16 19:19:18.176103 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:18.176084    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7\": container with ID starting with b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7 not found: ID does not exist" containerID="b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"
Sep 16 19:19:18.176174 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.176114    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7"} err="failed to get container status \"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7\": rpc error: code = NotFound desc = could not find container \"b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7\": container with ID starting with b18510756ca8ea4e5b3b548d56d9637e1be90de8852ef69a451fc7a2ee239fe7 not found: ID does not exist"
Sep 16 19:19:18.176174 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.176136    2563 scope.go:117] "RemoveContainer" containerID="83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"
Sep 16 19:19:18.176398 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:18.176374    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421\": container with ID starting with 83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421 not found: ID does not exist" containerID="83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"
Sep 16 19:19:18.176436 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.176403    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421"} err="failed to get container status \"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421\": rpc error: code = NotFound desc = could not find container \"83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421\": container with ID starting with 83ed7cf406f8bbda13f98a36d453192430dfc3d6281f92b5c584f6e7d16c3421 not found: ID does not exist"
Sep 16 19:19:18.182383 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.182365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:19:18.185910 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.185890    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a10-predictor-86b75c99c-66zjg"]
Sep 16 19:19:18.212928 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.212906    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:18.213048 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.212932    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b66c1c9-85af-4ec4-a37b-9ac29daca655-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:18.213048 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:18.212948    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4njp\" (UniqueName: \"kubernetes.io/projected/9b66c1c9-85af-4ec4-a37b-9ac29daca655-kube-api-access-f4njp\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.336728 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:19.336689    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:19.722503 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:19.722471    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" path="/var/lib/kubelet/pods/9b66c1c9-85af-4ec4-a37b-9ac29daca655/volumes"
Sep 16 19:19:23.164094 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:23.164066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:19:23.164644 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:23.164610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:23.165201 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:23.165185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:24.336505 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:24.336468    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:25.491549 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.491470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:25.491943 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.491644    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" containerID="cri-o://070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f" gracePeriod=30
Sep 16 19:19:25.521572 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.521548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.590111 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.590083    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:19:25.590315 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.590291    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container" containerID="cri-o://b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d" gracePeriod=30
Sep 16 19:19:25.590400 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.590336    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kube-rbac-proxy" containerID="cri-o://f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b" gracePeriod=30
Sep 16 19:19:25.642808 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.642780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.715325 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715297    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:19:25.715637 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715621    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kube-rbac-proxy"
Sep 16 19:19:25.715713 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kube-rbac-proxy"
Sep 16 19:19:25.715713 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container"
Sep 16 19:19:25.715713 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container"
Sep 16 19:19:25.715887 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715765    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kube-rbac-proxy"
Sep 16 19:19:25.715887 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.715779    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b66c1c9-85af-4ec4-a37b-9ac29daca655" containerName="kserve-container"
Sep 16 19:19:25.718938 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.718917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.721920 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.721903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-17a4d-predictor-serving-cert\""
Sep 16 19:19:25.721920 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.721913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-17a4d-kube-rbac-proxy-sar-config\""
Sep 16 19:19:25.728900 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.728880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:19:25.755588 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.755531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.853256 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.853225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.866949 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.866922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvl4\" (UniqueName: \"kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.867073 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.866977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.867073 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.867021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.968027 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.967999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvl4\" (UniqueName: \"kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.968144 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.968050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.968144 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.968076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.968244 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:25.968204    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-serving-cert: secret "success-200-isvc-17a4d-predictor-serving-cert" not found
Sep 16 19:19:25.968279 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:25.968269    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls podName:730ef7f6-6f92-412d-8346-bcd0ca6c3264 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:26.468254603 +0000 UTC m=+2055.206830084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls") pod "success-200-isvc-17a4d-predictor-6749977b8c-bxntq" (UID: "730ef7f6-6f92-412d-8346-bcd0ca6c3264") : secret "success-200-isvc-17a4d-predictor-serving-cert" not found
Sep 16 19:19:25.968723 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.968705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:25.976582 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:25.976562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvl4\" (UniqueName: \"kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:26.055718 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.055659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:26.155332 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.155308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:26.191571 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.191548    2563 generic.go:356] "Generic (PLEG): container finished" podID="6d7380ca-84ba-43db-be17-cc87934737ac" containerID="f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b" exitCode=2
Sep 16 19:19:26.191672 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.191618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerDied","Data":"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"}
Sep 16 19:19:26.472078 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.472047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:26.474401 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.474382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") pod \"success-200-isvc-17a4d-predictor-6749977b8c-bxntq\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") " pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:26.629306 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.629280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:26.898586 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:26.898559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:19:26.900267 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:19:26.900239    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730ef7f6_6f92_412d_8346_bcd0ca6c3264.slice/crio-8d5e59f6f89de217c43085604c0b007c913476d99ead2b958c3bf4267950ad5f WatchSource:0}: Error finding container 8d5e59f6f89de217c43085604c0b007c913476d99ead2b958c3bf4267950ad5f: Status 404 returned error can't find the container with id 8d5e59f6f89de217c43085604c0b007c913476d99ead2b958c3bf4267950ad5f
Sep 16 19:19:27.198037 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.197957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerStarted","Data":"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"}
Sep 16 19:19:27.198037 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.198013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerStarted","Data":"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"}
Sep 16 19:19:27.198037 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.198033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerStarted","Data":"8d5e59f6f89de217c43085604c0b007c913476d99ead2b958c3bf4267950ad5f"}
Sep 16 19:19:27.198300 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.198056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:27.198300 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.198076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:27.199371 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.199344    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:27.221044 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:27.220993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podStartSLOduration=2.220979436 podStartE2EDuration="2.220979436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:27.21987862 +0000 UTC m=+2055.958454117" watchObservedRunningTime="2026-09-16 19:19:27.220979436 +0000 UTC m=+2055.959554934"
Sep 16 19:19:28.165220 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.165175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:28.201761 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.201725    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:28.542853 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.542829    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:19:28.687912 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.687887    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls\") pod \"6d7380ca-84ba-43db-be17-cc87934737ac\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") "
Sep 16 19:19:28.688042 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.687935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2vfb\" (UniqueName: \"kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb\") pod \"6d7380ca-84ba-43db-be17-cc87934737ac\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") "
Sep 16 19:19:28.688042 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.687968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config\") pod \"6d7380ca-84ba-43db-be17-cc87934737ac\" (UID: \"6d7380ca-84ba-43db-be17-cc87934737ac\") "
Sep 16 19:19:28.688323 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.688293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d7380ca-84ba-43db-be17-cc87934737ac" (UID: "6d7380ca-84ba-43db-be17-cc87934737ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:28.689829 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.689799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d7380ca-84ba-43db-be17-cc87934737ac" (UID: "6d7380ca-84ba-43db-be17-cc87934737ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:28.689968 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.689943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb" (OuterVolumeSpecName: "kube-api-access-g2vfb") pod "6d7380ca-84ba-43db-be17-cc87934737ac" (UID: "6d7380ca-84ba-43db-be17-cc87934737ac"). InnerVolumeSpecName "kube-api-access-g2vfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:28.788525 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.788471    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2vfb\" (UniqueName: \"kubernetes.io/projected/6d7380ca-84ba-43db-be17-cc87934737ac-kube-api-access-g2vfb\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.788525 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.788493    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d7380ca-84ba-43db-be17-cc87934737ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:28.788525 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:28.788503    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d7380ca-84ba-43db-be17-cc87934737ac-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.206904 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.206868    2563 generic.go:356] "Generic (PLEG): container finished" podID="6d7380ca-84ba-43db-be17-cc87934737ac" containerID="b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d" exitCode=0
Sep 16 19:19:29.207294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.206920    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"
Sep 16 19:19:29.207294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.206950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerDied","Data":"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"}
Sep 16 19:19:29.207294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.207007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj" event={"ID":"6d7380ca-84ba-43db-be17-cc87934737ac","Type":"ContainerDied","Data":"1aa5800c648420038ca864972715e392ca17f7082ebb91eab08bb1e951ada80f"}
Sep 16 19:19:29.207294 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.207034    2563 scope.go:117] "RemoveContainer" containerID="f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"
Sep 16 19:19:29.215264 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.215246    2563 scope.go:117] "RemoveContainer" containerID="b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"
Sep 16 19:19:29.222280 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.222264    2563 scope.go:117] "RemoveContainer" containerID="f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"
Sep 16 19:19:29.222555 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:29.222538    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b\": container with ID starting with f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b not found: ID does not exist" containerID="f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"
Sep 16 19:19:29.222626 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.222561    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b"} err="failed to get container status \"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b\": rpc error: code = NotFound desc = could not find container \"f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b\": container with ID starting with f534d748e64c0860dc4714ff1dbce3af4c558375bf70ad930d80e1c078dbb43b not found: ID does not exist"
Sep 16 19:19:29.222626 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.222576    2563 scope.go:117] "RemoveContainer" containerID="b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"
Sep 16 19:19:29.222828 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:29.222798    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d\": container with ID starting with b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d not found: ID does not exist" containerID="b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"
Sep 16 19:19:29.222881 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.222841    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d"} err="failed to get container status \"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d\": rpc error: code = NotFound desc = could not find container \"b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d\": container with ID starting with b62a9bb1ae35fc0022a430a731e4b622b21bea09017cdefaf680dc1897a2284d not found: ID does not exist"
Sep 16 19:19:29.230762 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.230740    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:19:29.234423 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.234391    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bba6-predictor-65498c7465-jfhbj"]
Sep 16 19:19:29.337013 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.336984    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:29.337119 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.337104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:19:29.722409 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:29.722368    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" path="/var/lib/kubelet/pods/6d7380ca-84ba-43db-be17-cc87934737ac/volumes"
Sep 16 19:19:33.164277 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:33.164236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:33.164700 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:33.164527    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:33.205609 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:33.205587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:19:33.206004 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:33.205979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:34.337206 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:34.337105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:38.164596 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:38.164557    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:38.165009 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:38.164661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:39.336887 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:39.336850    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:40.805067 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:40.805035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:43.164913 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:43.164873    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:43.165384 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:43.165200    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:43.206473 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:43.206439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:44.336385 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:44.336350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:45.121212 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.121181    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:19:45.203721 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.203696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") pod \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") "
Sep 16 19:19:45.203872 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.203741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle\") pod \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\" (UID: \"d6c2b4d2-499b-4975-901d-7a6f28e71ab0\") "
Sep 16 19:19:45.204115 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.204089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d6c2b4d2-499b-4975-901d-7a6f28e71ab0" (UID: "d6c2b4d2-499b-4975-901d-7a6f28e71ab0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:45.205616 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.205598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6c2b4d2-499b-4975-901d-7a6f28e71ab0" (UID: "d6c2b4d2-499b-4975-901d-7a6f28e71ab0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:45.264724 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.264656    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerID="d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078" exitCode=0
Sep 16 19:19:45.264724 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.264700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"
Sep 16 19:19:45.264724 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.264714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" event={"ID":"d6c2b4d2-499b-4975-901d-7a6f28e71ab0","Type":"ContainerDied","Data":"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"}
Sep 16 19:19:45.264936 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.264766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp" event={"ID":"d6c2b4d2-499b-4975-901d-7a6f28e71ab0","Type":"ContainerDied","Data":"9d2d206d40d4e146518d7054e431e0be67e88b062f88c3a5a91133b008058790"}
Sep 16 19:19:45.264936 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.264795    2563 scope.go:117] "RemoveContainer" containerID="d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"
Sep 16 19:19:45.272658 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.272644    2563 scope.go:117] "RemoveContainer" containerID="d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"
Sep 16 19:19:45.272947 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:45.272927    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078\": container with ID starting with d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078 not found: ID does not exist" containerID="d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"
Sep 16 19:19:45.273001 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.272956    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078"} err="failed to get container status \"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078\": rpc error: code = NotFound desc = could not find container \"d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078\": container with ID starting with d08e14a992d5cdea5fbd5fb4a88b2be12ac297a20ccff99cb25070ade61d1078 not found: ID does not exist"
Sep 16 19:19:45.304512 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.304490    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:45.304512 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.304513    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6c2b4d2-499b-4975-901d-7a6f28e71ab0-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:45.306503 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.306475    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:19:45.309321 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.309300    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a10-756f44fbbb-hpvpp"]
Sep 16 19:19:45.723032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:45.723004    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" path="/var/lib/kubelet/pods/d6c2b4d2-499b-4975-901d-7a6f28e71ab0/volumes"
Sep 16 19:19:48.164923 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:48.164863    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:53.164262 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:53.164225    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:19:53.164650 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:53.164530    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 19:19:53.206631 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:53.206599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:19:55.629112 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.629085    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:55.779590 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.779527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") pod \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") "
Sep 16 19:19:55.779737 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.779599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle\") pod \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\" (UID: \"7154dd1e-7dd6-4a7d-b671-6ed9d1776058\") "
Sep 16 19:19:55.779899 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.779871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7154dd1e-7dd6-4a7d-b671-6ed9d1776058" (UID: "7154dd1e-7dd6-4a7d-b671-6ed9d1776058"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:55.781514 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.781495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7154dd1e-7dd6-4a7d-b671-6ed9d1776058" (UID: "7154dd1e-7dd6-4a7d-b671-6ed9d1776058"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:55.880689 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.880664    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.880689 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:55.880688    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7154dd1e-7dd6-4a7d-b671-6ed9d1776058-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:19:56.308723 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.308689    2563 generic.go:356] "Generic (PLEG): container finished" podID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerID="070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f" exitCode=0
Sep 16 19:19:56.308946 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.308733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"
Sep 16 19:19:56.308946 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.308777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" event={"ID":"7154dd1e-7dd6-4a7d-b671-6ed9d1776058","Type":"ContainerDied","Data":"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"}
Sep 16 19:19:56.308946 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.308853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b" event={"ID":"7154dd1e-7dd6-4a7d-b671-6ed9d1776058","Type":"ContainerDied","Data":"e49baa1a115dfcdb7ef784c5369cfbff2c4bab66f83148e96f54b9c146be35d2"}
Sep 16 19:19:56.308946 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.308888    2563 scope.go:117] "RemoveContainer" containerID="070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"
Sep 16 19:19:56.317119 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.317105    2563 scope.go:117] "RemoveContainer" containerID="070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"
Sep 16 19:19:56.317391 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:19:56.317372    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f\": container with ID starting with 070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f not found: ID does not exist" containerID="070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"
Sep 16 19:19:56.317463 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.317398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f"} err="failed to get container status \"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f\": rpc error: code = NotFound desc = could not find container \"070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f\": container with ID starting with 070800d63015b5f270a235bead33b85ce5dc66af1dca39c450e957e9d9510b5f not found: ID does not exist"
Sep 16 19:19:56.330575 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.330554    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:56.333994 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:56.333974    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3bba6-879cd4d78-6rx4b"]
Sep 16 19:19:57.722327 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:19:57.722293    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" path="/var/lib/kubelet/pods/7154dd1e-7dd6-4a7d-b671-6ed9d1776058/volumes"
Sep 16 19:20:03.164948 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:03.164920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:20:03.206082 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:03.206046    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:20:13.206979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:13.206949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:20:15.239616 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.239571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240030    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240068    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240101    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240110    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240118    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kube-rbac-proxy"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240127    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kube-rbac-proxy"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240195    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kserve-container"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c2b4d2-499b-4975-901d-7a6f28e71ab0" containerName="sequence-graph-53a10"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240217    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7380ca-84ba-43db-be17-cc87934737ac" containerName="kube-rbac-proxy"
Sep 16 19:20:15.240243 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.240226    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7154dd1e-7dd6-4a7d-b671-6ed9d1776058" containerName="splitter-graph-3bba6"
Sep 16 19:20:15.244692 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.244668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.247573 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.247550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7f5a4-kube-rbac-proxy-sar-config\""
Sep 16 19:20:15.247685 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.247550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7f5a4-serving-cert\""
Sep 16 19:20:15.248999 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.248977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:15.253280 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.253259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:20:15.377690 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.377658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:15.415290 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.415267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.415382 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.415298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.515591 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.515516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.515591 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.515558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.516296 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.516269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.518256 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.518231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls\") pod \"switch-graph-7f5a4-6d85f7cc45-dh59j\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") " pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.555402 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.555368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:15.827869 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:15.827846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:20:15.830217 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:20:15.830181    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7f486c_5298_4b04_963d_f6de588ca1f2.slice/crio-41868d9b9162c4e2711909517ceba9bfe57d879ece3fc201b13af2fb7cd3434c WatchSource:0}: Error finding container 41868d9b9162c4e2711909517ceba9bfe57d879ece3fc201b13af2fb7cd3434c: Status 404 returned error can't find the container with id 41868d9b9162c4e2711909517ceba9bfe57d879ece3fc201b13af2fb7cd3434c
Sep 16 19:20:16.384203 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:16.384170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" event={"ID":"0d7f486c-5298-4b04-963d-f6de588ca1f2","Type":"ContainerStarted","Data":"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"}
Sep 16 19:20:16.384576 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:16.384241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" event={"ID":"0d7f486c-5298-4b04-963d-f6de588ca1f2","Type":"ContainerStarted","Data":"41868d9b9162c4e2711909517ceba9bfe57d879ece3fc201b13af2fb7cd3434c"}
Sep 16 19:20:16.384576 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:16.384278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:16.401931 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:16.401888    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podStartSLOduration=1.401873951 podStartE2EDuration="1.401873951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:16.401589163 +0000 UTC m=+2105.140164663" watchObservedRunningTime="2026-09-16 19:20:16.401873951 +0000 UTC m=+2105.140449452"
Sep 16 19:20:22.393343 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:22.393305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:20:25.747730 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.747702    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:20:25.751071 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.751051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.755020 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.754990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-17a4d-kube-rbac-proxy-sar-config\""
Sep 16 19:20:25.755020 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.755010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-17a4d-serving-cert\""
Sep 16 19:20:25.760485 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.760455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:25.767867 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.767843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:20:25.783060 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.783032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.783425 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.783404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.883680 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.883658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.883774 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.883703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.884383 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.884354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.886032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.886012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls\") pod \"splitter-graph-17a4d-6c9ff65448-jm9m7\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") " pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:25.917048 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:25.917008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:26.060344 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.060284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:26.334181 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.334116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:20:26.337850 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:20:26.337803    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1a25f0_8e32_407c_83ed_a6112d4c7a6d.slice/crio-08d65feed8a58486cf01b5e2810c3a2bc6f131f0fb15a1224941822e9c7bfa1c WatchSource:0}: Error finding container 08d65feed8a58486cf01b5e2810c3a2bc6f131f0fb15a1224941822e9c7bfa1c: Status 404 returned error can't find the container with id 08d65feed8a58486cf01b5e2810c3a2bc6f131f0fb15a1224941822e9c7bfa1c
Sep 16 19:20:26.421880 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.421849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" event={"ID":"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d","Type":"ContainerStarted","Data":"8ed32dff8ba64d2d783e029d826279ba2dca019b205153ea2a2503d5575995e7"}
Sep 16 19:20:26.421991 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.421905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" event={"ID":"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d","Type":"ContainerStarted","Data":"08d65feed8a58486cf01b5e2810c3a2bc6f131f0fb15a1224941822e9c7bfa1c"}
Sep 16 19:20:26.422040 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.422024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:20:26.439652 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:26.439614    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podStartSLOduration=1.439599185 podStartE2EDuration="1.439599185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:26.439021767 +0000 UTC m=+2115.177597266" watchObservedRunningTime="2026-09-16 19:20:26.439599185 +0000 UTC m=+2115.178174686"
Sep 16 19:20:32.431416 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:20:32.431377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:21:01.841183 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:21:01.841143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:21.184601 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:21:21.184506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:34.996737 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:22:34.996644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:35.104337 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:22:35.104292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:06.594641 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:23:06.594595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:26.181705 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:24:26.181625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:05.343042 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:25:05.342996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:40.248166 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.248090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:28:40.248702 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.248298    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" containerID="cri-o://8ed32dff8ba64d2d783e029d826279ba2dca019b205153ea2a2503d5575995e7" gracePeriod=30
Sep 16 19:28:40.314850 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.314791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:40.372843 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.372801    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:28:40.373091 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.373059    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container" containerID="cri-o://03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000" gracePeriod=30
Sep 16 19:28:40.373232 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.373099    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kube-rbac-proxy" containerID="cri-o://2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605" gracePeriod=30
Sep 16 19:28:40.418556 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.418528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:40.513267 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:40.513211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:41.244931 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:41.244894    2563 generic.go:356] "Generic (PLEG): container finished" podID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerID="2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605" exitCode=2
Sep 16 19:28:41.245133 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:41.244960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerDied","Data":"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"}
Sep 16 19:28:42.429953 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:42.429916    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:28:42.908026 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:42.908003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:28:42.999773 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:42.999714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zvl4\" (UniqueName: \"kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4\") pod \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") "
Sep 16 19:28:42.999773 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:42.999741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config\") pod \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") "
Sep 16 19:28:42.999926 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:42.999793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") pod \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\" (UID: \"730ef7f6-6f92-412d-8346-bcd0ca6c3264\") "
Sep 16 19:28:43.000169 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.000149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "730ef7f6-6f92-412d-8346-bcd0ca6c3264" (UID: "730ef7f6-6f92-412d-8346-bcd0ca6c3264"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:43.001829 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.001795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4" (OuterVolumeSpecName: "kube-api-access-7zvl4") pod "730ef7f6-6f92-412d-8346-bcd0ca6c3264" (UID: "730ef7f6-6f92-412d-8346-bcd0ca6c3264"). InnerVolumeSpecName "kube-api-access-7zvl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:43.001896 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.001804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "730ef7f6-6f92-412d-8346-bcd0ca6c3264" (UID: "730ef7f6-6f92-412d-8346-bcd0ca6c3264"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:43.105932 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.101676    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvl4\" (UniqueName: \"kubernetes.io/projected/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-api-access-7zvl4\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.105932 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.101705    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730ef7f6-6f92-412d-8346-bcd0ca6c3264-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.105932 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.101728    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730ef7f6-6f92-412d-8346-bcd0ca6c3264-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.254218 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.254149    2563 generic.go:356] "Generic (PLEG): container finished" podID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerID="03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000" exitCode=0
Sep 16 19:28:43.254218 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.254212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"
Sep 16 19:28:43.254397 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.254245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerDied","Data":"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"}
Sep 16 19:28:43.254397 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.254352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq" event={"ID":"730ef7f6-6f92-412d-8346-bcd0ca6c3264","Type":"ContainerDied","Data":"8d5e59f6f89de217c43085604c0b007c913476d99ead2b958c3bf4267950ad5f"}
Sep 16 19:28:43.254504 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.254397    2563 scope.go:117] "RemoveContainer" containerID="2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"
Sep 16 19:28:43.263121 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.263098    2563 scope.go:117] "RemoveContainer" containerID="03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"
Sep 16 19:28:43.270152 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.270137    2563 scope.go:117] "RemoveContainer" containerID="2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"
Sep 16 19:28:43.270414 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:28:43.270399    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605\": container with ID starting with 2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605 not found: ID does not exist" containerID="2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"
Sep 16 19:28:43.270466 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.270419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605"} err="failed to get container status \"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605\": rpc error: code = NotFound desc = could not find container \"2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605\": container with ID starting with 2d18eb5edf23aa8f082e76c8b672acd4adc046893d968e089056e73bbf65f605 not found: ID does not exist"
Sep 16 19:28:43.270466 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.270434    2563 scope.go:117] "RemoveContainer" containerID="03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"
Sep 16 19:28:43.270653 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:28:43.270635    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000\": container with ID starting with 03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000 not found: ID does not exist" containerID="03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"
Sep 16 19:28:43.270690 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.270659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000"} err="failed to get container status \"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000\": rpc error: code = NotFound desc = could not find container \"03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000\": container with ID starting with 03a86d77d9ae4a098690535656f4766782e796aaae6a281eb2d16b2712882000 not found: ID does not exist"
Sep 16 19:28:43.278192 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.278173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:28:43.284466 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.284445    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-17a4d-predictor-6749977b8c-bxntq"]
Sep 16 19:28:43.722652 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:43.722627    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" path="/var/lib/kubelet/pods/730ef7f6-6f92-412d-8346-bcd0ca6c3264/volumes"
Sep 16 19:28:47.430119 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:47.430078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:28:52.430363 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:52.430324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:28:52.430741 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:52.430423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:28:57.430246 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:28:57.430211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:29:02.430178 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:02.430140    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:29:07.429581 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:07.429532    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:29:10.349949 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.349917    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerID="8ed32dff8ba64d2d783e029d826279ba2dca019b205153ea2a2503d5575995e7" exitCode=0
Sep 16 19:29:10.350258 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.349969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" event={"ID":"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d","Type":"ContainerDied","Data":"8ed32dff8ba64d2d783e029d826279ba2dca019b205153ea2a2503d5575995e7"}
Sep 16 19:29:10.386344 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.386323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:29:10.485962 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.485933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle\") pod \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") "
Sep 16 19:29:10.486113 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.485982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls\") pod \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\" (UID: \"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d\") "
Sep 16 19:29:10.486210 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.486187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" (UID: "7f1a25f0-8e32-407c-83ed-a6112d4c7a6d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:10.487943 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.487916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" (UID: "7f1a25f0-8e32-407c-83ed-a6112d4c7a6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:10.586494 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.586439    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:29:10.586494 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:10.586460    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:29:11.355648 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.355617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"
Sep 16 19:29:11.356265 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.355640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7" event={"ID":"7f1a25f0-8e32-407c-83ed-a6112d4c7a6d","Type":"ContainerDied","Data":"08d65feed8a58486cf01b5e2810c3a2bc6f131f0fb15a1224941822e9c7bfa1c"}
Sep 16 19:29:11.356265 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.355706    2563 scope.go:117] "RemoveContainer" containerID="8ed32dff8ba64d2d783e029d826279ba2dca019b205153ea2a2503d5575995e7"
Sep 16 19:29:11.381427 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.381391    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:29:11.388179 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.388151    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-17a4d-6c9ff65448-jm9m7"]
Sep 16 19:29:11.722575 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:29:11.722550    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" path="/var/lib/kubelet/pods/7f1a25f0-8e32-407c-83ed-a6112d4c7a6d/volumes"
Sep 16 19:31:42.374751 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:31:42.374664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:20.712786 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:32:20.712752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:34.240463 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.240389    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:36:34.241005 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.240589    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" containerID="cri-o://d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b" gracePeriod=30
Sep 16 19:36:34.297959 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.297923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:34.383057 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.383029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:36:34.383268 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.383228    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container" containerID="cri-o://51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab" gracePeriod=30
Sep 16 19:36:34.383400 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.383265    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kube-rbac-proxy" containerID="cri-o://20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058" gracePeriod=30
Sep 16 19:36:34.412576 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.412546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:34.513184 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.513160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:34.917289 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.917259    2563 generic.go:356] "Generic (PLEG): container finished" podID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerID="20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058" exitCode=2
Sep 16 19:36:34.917471 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:34.917305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerDied","Data":"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"}
Sep 16 19:36:35.906225 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:35.906181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:36.001772 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.001744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:36.706446 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.706424    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:36:36.818516 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.818447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffjjp\" (UniqueName: \"kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp\") pod \"743bd70b-3137-4d57-8c52-9c019a6aa05d\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") "
Sep 16 19:36:36.818516 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.818502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config\") pod \"743bd70b-3137-4d57-8c52-9c019a6aa05d\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") "
Sep 16 19:36:36.818680 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.818534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") pod \"743bd70b-3137-4d57-8c52-9c019a6aa05d\" (UID: \"743bd70b-3137-4d57-8c52-9c019a6aa05d\") "
Sep 16 19:36:36.818877 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.818847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "743bd70b-3137-4d57-8c52-9c019a6aa05d" (UID: "743bd70b-3137-4d57-8c52-9c019a6aa05d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:36.820481 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.820452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp" (OuterVolumeSpecName: "kube-api-access-ffjjp") pod "743bd70b-3137-4d57-8c52-9c019a6aa05d" (UID: "743bd70b-3137-4d57-8c52-9c019a6aa05d"). InnerVolumeSpecName "kube-api-access-ffjjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:36.820481 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.820472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "743bd70b-3137-4d57-8c52-9c019a6aa05d" (UID: "743bd70b-3137-4d57-8c52-9c019a6aa05d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:36.919480 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.919453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:36:36.919480 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.919477    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/743bd70b-3137-4d57-8c52-9c019a6aa05d-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:36:36.919834 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.919491    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjjp\" (UniqueName: \"kubernetes.io/projected/743bd70b-3137-4d57-8c52-9c019a6aa05d-kube-api-access-ffjjp\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:36:36.925224 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.925201    2563 generic.go:356] "Generic (PLEG): container finished" podID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerID="51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab" exitCode=0
Sep 16 19:36:36.925349 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.925232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerDied","Data":"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"}
Sep 16 19:36:36.925349 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.925265    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"
Sep 16 19:36:36.925349 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.925277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd" event={"ID":"743bd70b-3137-4d57-8c52-9c019a6aa05d","Type":"ContainerDied","Data":"cdb4972e056c57548b2d2957ad5c4e007ae3e51ea838a98b28467c169d7bd9fd"}
Sep 16 19:36:36.925349 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.925302    2563 scope.go:117] "RemoveContainer" containerID="20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"
Sep 16 19:36:36.933714 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.933698    2563 scope.go:117] "RemoveContainer" containerID="51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"
Sep 16 19:36:36.940357 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.940341    2563 scope.go:117] "RemoveContainer" containerID="20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"
Sep 16 19:36:36.940617 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:36:36.940598    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058\": container with ID starting with 20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058 not found: ID does not exist" containerID="20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"
Sep 16 19:36:36.940671 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.940624    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058"} err="failed to get container status \"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058\": rpc error: code = NotFound desc = could not find container \"20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058\": container with ID starting with 20b39c71b4e74ba829be10a0c2f3f90c7d473be41fb895c12b3669a0933b4058 not found: ID does not exist"
Sep 16 19:36:36.940671 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.940640    2563 scope.go:117] "RemoveContainer" containerID="51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"
Sep 16 19:36:36.940893 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:36:36.940863    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab\": container with ID starting with 51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab not found: ID does not exist" containerID="51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"
Sep 16 19:36:36.940977 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.940895    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab"} err="failed to get container status \"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab\": rpc error: code = NotFound desc = could not find container \"51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab\": container with ID starting with 51d36cdface119e78e7d984fa0e3c593c5054b62f973fba92f296d36603acbab not found: ID does not exist"
Sep 16 19:36:36.947637 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.947617    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:36:36.952076 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:36.952056    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f5a4-predictor-85677f6c9f-gq7wd"]
Sep 16 19:36:37.391464 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:37.391428    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:36:37.725302 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:37.725216    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" path="/var/lib/kubelet/pods/743bd70b-3137-4d57-8c52-9c019a6aa05d/volumes"
Sep 16 19:36:42.391843 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:42.391796    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:36:47.392002 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:47.391965    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:36:47.392461 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:47.392083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:36:51.967675 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:51.967643    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:52.391032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:52.391000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:36:52.810075 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:52.809993    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:53.654482 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:53.654451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:54.489654 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:54.486128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:55.306747 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:55.306721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:56.110444 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:56.110408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:56.934588 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:56.934559    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:57.391291 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:57.391251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:36:57.730016 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:57.729949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:58.527948 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:58.527918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:36:59.335557 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:36:59.335523    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:00.145846 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:00.145798    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:00.943715 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:00.943685    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:01.739936 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:01.739907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:02.391190 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:02.391156    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:37:02.560574 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:02.560542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:03.427061 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:03.427012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f5a4-6d85f7cc45-dh59j_0d7f486c-5298-4b04-963d-f6de588ca1f2/switch-graph-7f5a4/0.log"
Sep 16 19:37:04.380384 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.380356    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:37:04.502031 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.501970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls\") pod \"0d7f486c-5298-4b04-963d-f6de588ca1f2\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") "
Sep 16 19:37:04.502338 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.502031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle\") pod \"0d7f486c-5298-4b04-963d-f6de588ca1f2\" (UID: \"0d7f486c-5298-4b04-963d-f6de588ca1f2\") "
Sep 16 19:37:04.502383 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.502333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d7f486c-5298-4b04-963d-f6de588ca1f2" (UID: "0d7f486c-5298-4b04-963d-f6de588ca1f2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:04.503908 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.503885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d7f486c-5298-4b04-963d-f6de588ca1f2" (UID: "0d7f486c-5298-4b04-963d-f6de588ca1f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:04.603045 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.603021    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7f486c-5298-4b04-963d-f6de588ca1f2-openshift-service-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:37:04.603045 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:04.603040    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d7f486c-5298-4b04-963d-f6de588ca1f2-proxy-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\""
Sep 16 19:37:05.019002 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.018973    2563 generic.go:356] "Generic (PLEG): container finished" podID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerID="d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b" exitCode=0
Sep 16 19:37:05.019170 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.019011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" event={"ID":"0d7f486c-5298-4b04-963d-f6de588ca1f2","Type":"ContainerDied","Data":"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"}
Sep 16 19:37:05.019170 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.019038    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"
Sep 16 19:37:05.019170 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.019051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j" event={"ID":"0d7f486c-5298-4b04-963d-f6de588ca1f2","Type":"ContainerDied","Data":"41868d9b9162c4e2711909517ceba9bfe57d879ece3fc201b13af2fb7cd3434c"}
Sep 16 19:37:05.019170 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.019078    2563 scope.go:117] "RemoveContainer" containerID="d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"
Sep 16 19:37:05.028799 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.028774    2563 scope.go:117] "RemoveContainer" containerID="d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"
Sep 16 19:37:05.029117 ip-10-0-137-226 kubenswrapper[2563]: E0916 19:37:05.029095    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b\": container with ID starting with d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b not found: ID does not exist" containerID="d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"
Sep 16 19:37:05.029157 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.029126    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b"} err="failed to get container status \"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b\": rpc error: code = NotFound desc = could not find container \"d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b\": container with ID starting with d0e28296f39e9bcddf8ee30a2f5df3faac1974e1ddf0958f822ddec16d7e970b not found: ID does not exist"
Sep 16 19:37:05.041241 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.041214    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:37:05.047302 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.047278    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f5a4-6d85f7cc45-dh59j"]
Sep 16 19:37:05.723026 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:05.722995    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" path="/var/lib/kubelet/pods/0d7f486c-5298-4b04-963d-f6de588ca1f2/volumes"
Sep 16 19:37:07.267681 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.267631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:07.406350 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406319    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grqt8/must-gather-xhjsf"]
Sep 16 19:37:07.406550 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406538    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406562    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406581    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406590    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406604    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406609    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406611 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406615    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406621    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406663    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kserve-container"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406675    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7f486c-5298-4b04-963d-f6de588ca1f2" containerName="switch-graph-7f5a4"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406680    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="743bd70b-3137-4d57-8c52-9c019a6aa05d" containerName="kube-rbac-proxy"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406685    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1a25f0-8e32-407c-83ed-a6112d4c7a6d" containerName="splitter-graph-17a4d"
Sep 16 19:37:07.406979 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.406690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="730ef7f6-6f92-412d-8346-bcd0ca6c3264" containerName="kserve-container"
Sep 16 19:37:07.410961 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.410945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.413805 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.413786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"openshift-service-ca.crt\""
Sep 16 19:37:07.413943 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.413870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grqt8\"/\"default-dockercfg-k5d84\""
Sep 16 19:37:07.415202 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.415184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"kube-root-ca.crt\""
Sep 16 19:37:07.422715 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.422684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/must-gather-xhjsf"]
Sep 16 19:37:07.524062 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.524003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/805a8df8-78eb-4070-be54-cf6013a1a089-must-gather-output\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.524062 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.524042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2v4\" (UniqueName: \"kubernetes.io/projected/805a8df8-78eb-4070-be54-cf6013a1a089-kube-api-access-kv2v4\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.624573 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.624549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2v4\" (UniqueName: \"kubernetes.io/projected/805a8df8-78eb-4070-be54-cf6013a1a089-kube-api-access-kv2v4\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.624652 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.624597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/805a8df8-78eb-4070-be54-cf6013a1a089-must-gather-output\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.624877 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.624862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/805a8df8-78eb-4070-be54-cf6013a1a089-must-gather-output\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.633848 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.633809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2v4\" (UniqueName: \"kubernetes.io/projected/805a8df8-78eb-4070-be54-cf6013a1a089-kube-api-access-kv2v4\") pod \"must-gather-xhjsf\" (UID: \"805a8df8-78eb-4070-be54-cf6013a1a089\") " pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.719497 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.719474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/must-gather-xhjsf"
Sep 16 19:37:07.990448 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.990419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/must-gather-xhjsf"]
Sep 16 19:37:07.993059 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:37:07.993035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod805a8df8_78eb_4070_be54_cf6013a1a089.slice/crio-e0fd81880c789e36563ec014badd44b575d2d124025ace53b80313c3074e911d WatchSource:0}: Error finding container e0fd81880c789e36563ec014badd44b575d2d124025ace53b80313c3074e911d: Status 404 returned error can't find the container with id e0fd81880c789e36563ec014badd44b575d2d124025ace53b80313c3074e911d
Sep 16 19:37:07.994754 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:07.994739    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:37:08.030950 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:08.030924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/must-gather-xhjsf" event={"ID":"805a8df8-78eb-4070-be54-cf6013a1a089","Type":"ContainerStarted","Data":"e0fd81880c789e36563ec014badd44b575d2d124025ace53b80313c3074e911d"}
Sep 16 19:37:08.565459 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:08.565427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:09.038846 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:09.038794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/must-gather-xhjsf" event={"ID":"805a8df8-78eb-4070-be54-cf6013a1a089","Type":"ContainerStarted","Data":"9f42c9cfad5c09de0d291506c580ab63db42e17144133bc86d69c56ce87cf522"}
Sep 16 19:37:09.039032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:09.038863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/must-gather-xhjsf" event={"ID":"805a8df8-78eb-4070-be54-cf6013a1a089","Type":"ContainerStarted","Data":"41a2ec71dfb9db93dd78981e9673854b2b5c3c73368ccaf623c8944457112499"}
Sep 16 19:37:09.058489 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:09.058443    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grqt8/must-gather-xhjsf" podStartSLOduration=1.369585024 podStartE2EDuration="2.058427167s" totalImagesPullingTime="688.842143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:09.056448559 +0000 UTC m=+3117.795024056" watchObservedRunningTime="2026-09-16 19:37:09.058427167 +0000 UTC m=+3117.797002667"
Sep 16 19:37:10.014610 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:10.014575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-57jlt_c3aa8d45-86ee-44c6-b9dc-01e878c909b5/global-pull-secret-syncer/0.log"
Sep 16 19:37:10.146933 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:10.146901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log"
Sep 16 19:37:10.261744 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:10.261714    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-226.ec2.internal_5bf20f76c0d057976dff6a847f7d0954/haproxy/0.log"
Sep 16 19:37:13.104161 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:13.104127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:13.203797 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:13.203755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-77tmj_1224ee06-367b-4d82-a884-2ad98cfaed41/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:14.417011 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:14.416981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/node-exporter/0.log"
Sep 16 19:37:14.447154 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:14.447117    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/kube-rbac-proxy/0.log"
Sep 16 19:37:14.474423 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:14.474397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d7gkw_f1667821-9b9d-4315-b29f-cda37b4c7fe2/init-textfile/0.log"
Sep 16 19:37:18.297257 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.297222    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"]
Sep 16 19:37:18.302719 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.302695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.312917 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.312895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"]
Sep 16 19:37:18.314202 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.314177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-sys\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.314317 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.314248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-podres\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.314317 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.314291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-proc\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.314399 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.314328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-lib-modules\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.314399 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.314379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqc7\" (UniqueName: \"kubernetes.io/projected/172b4584-bf33-4d6e-82bc-8491b9ab0925-kube-api-access-9fqc7\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.415179 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.415139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-proc\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.415428 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.415405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-lib-modules\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.415774 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.415736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqc7\" (UniqueName: \"kubernetes.io/projected/172b4584-bf33-4d6e-82bc-8491b9ab0925-kube-api-access-9fqc7\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416055 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.416028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-sys\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.416094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-podres\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.415660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-lib-modules\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416148 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.415354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-proc\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416317 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.416192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-sys\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.416317 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.416274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/172b4584-bf33-4d6e-82bc-8491b9ab0925-podres\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.427471 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.427415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqc7\" (UniqueName: \"kubernetes.io/projected/172b4584-bf33-4d6e-82bc-8491b9ab0925-kube-api-access-9fqc7\") pod \"perf-node-gather-daemonset-l4t7j\" (UID: \"172b4584-bf33-4d6e-82bc-8491b9ab0925\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.613973 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.613942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:18.899771 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:18.899746    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"]
Sep 16 19:37:18.901620 ip-10-0-137-226 kubenswrapper[2563]: W0916 19:37:18.901580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod172b4584_bf33_4d6e_82bc_8491b9ab0925.slice/crio-1b444f40d61f0a70d618646d1864b19746240e5ce8efad96896968dd464c8e93 WatchSource:0}: Error finding container 1b444f40d61f0a70d618646d1864b19746240e5ce8efad96896968dd464c8e93: Status 404 returned error can't find the container with id 1b444f40d61f0a70d618646d1864b19746240e5ce8efad96896968dd464c8e93
Sep 16 19:37:19.088721 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.088682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j" event={"ID":"172b4584-bf33-4d6e-82bc-8491b9ab0925","Type":"ContainerStarted","Data":"14c026a56afd8f3c184506a766081f8bf1192dec33d631c5c2483770ba35dff2"}
Sep 16 19:37:19.088929 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.088743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j" event={"ID":"172b4584-bf33-4d6e-82bc-8491b9ab0925","Type":"ContainerStarted","Data":"1b444f40d61f0a70d618646d1864b19746240e5ce8efad96896968dd464c8e93"}
Sep 16 19:37:19.088929 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.088803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:19.150040 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.149960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8bh4h_f0b6d71b-55f4-4abf-a549-b2e7a236b7b0/dns/0.log"
Sep 16 19:37:19.175407 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.175382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8bh4h_f0b6d71b-55f4-4abf-a549-b2e7a236b7b0/kube-rbac-proxy/0.log"
Sep 16 19:37:19.345334 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.345301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s8xph_14610c4a-a5e4-4bbc-a969-333868322d1c/dns-node-resolver/0.log"
Sep 16 19:37:19.873325 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:19.873300    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mq86t_0672e4fc-19cc-464f-9123-8b90a9ce0107/node-ca/0.log"
Sep 16 19:37:21.045452 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:21.045419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9sbcb_e4a58c97-7aae-48ff-b8c5-10a7948a853f/serve-healthcheck-canary/0.log"
Sep 16 19:37:21.419239 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:21.419214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fkgr_a9550f81-9761-40cd-a78d-68d965355415/kube-rbac-proxy/0.log"
Sep 16 19:37:21.442463 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:21.442442    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fkgr_a9550f81-9761-40cd-a78d-68d965355415/exporter/0.log"
Sep 16 19:37:21.466516 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:21.466497    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fkgr_a9550f81-9761-40cd-a78d-68d965355415/extractor/0.log"
Sep 16 19:37:24.013968 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:24.013910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-gl8lw_becdef2b-631f-4805-8359-deca750c8cb2/manager/0.log"
Sep 16 19:37:24.147143 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:24.147116    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2cx84_1b76fedc-477b-4d7e-b9f7-6b777e8427c0/seaweedfs/0.log"
Sep 16 19:37:25.102302 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:25.102268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j"
Sep 16 19:37:25.122161 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:25.122111    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-l4t7j" podStartSLOduration=7.122095841 podStartE2EDuration="7.122095841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:19.112325913 +0000 UTC m=+3127.850901423" watchObservedRunningTime="2026-09-16 19:37:25.122095841 +0000 UTC m=+3133.860671377"
Sep 16 19:37:30.148323 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.148293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:37:30.173347 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.173321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/egress-router-binary-copy/0.log"
Sep 16 19:37:30.197342 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.197325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/cni-plugins/0.log"
Sep 16 19:37:30.220564 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.220538    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/bond-cni-plugin/0.log"
Sep 16 19:37:30.245032 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.245006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/routeoverride-cni/0.log"
Sep 16 19:37:30.268965 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.268946    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/whereabouts-cni-bincopy/0.log"
Sep 16 19:37:30.292843 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.292811    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vznp7_2c6d9c30-89bc-4ca4-8202-f77f3871e8ae/whereabouts-cni/0.log"
Sep 16 19:37:30.356195 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.356178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w6zm4_9c0aa788-3d2f-4abb-9e00-72cb1181215f/kube-multus/0.log"
Sep 16 19:37:30.473833 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.473760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mmtgl_1dd2fed3-b044-4df8-9a04-0f1de72ac07c/network-metrics-daemon/0.log"
Sep 16 19:37:30.498001 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:30.497980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mmtgl_1dd2fed3-b044-4df8-9a04-0f1de72ac07c/kube-rbac-proxy/0.log"
Sep 16 19:37:31.626364 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.626336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/ovn-controller/0.log"
Sep 16 19:37:31.648132 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.648110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/ovn-acl-logging/0.log"
Sep 16 19:37:31.674511 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.674485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/kube-rbac-proxy-node/0.log"
Sep 16 19:37:31.698801 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.698780    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:37:31.720429 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.720404    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/northd/0.log"
Sep 16 19:37:31.743240 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.743217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/nbdb/0.log"
Sep 16 19:37:31.767381 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.767361    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/sbdb/0.log"
Sep 16 19:37:31.810028 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:31.810009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xdr6v_ae3c2102-ddc8-4558-b65c-77ec51194fcf/ovnkube-controller/0.log"
Sep 16 19:37:32.817001 ip-10-0-137-226 kubenswrapper[2563]: I0916 19:37:32.816971    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jmbzj_f9afdf29-14a2-48be-8732-7981a9c78d80/network-check-target-container/0.log"