Sep 17 23:08:41.316055 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:08:41.316064 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:08:41.316071 ip-10-0-128-70 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:08:41.316208 ip-10-0-128-70 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:08:51.445560 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:08:51.445575 ip-10-0-128-70 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a9a83832e55944f09289c477a907e7b7 --
Sep 17 23:10:43.745313 ip-10-0-128-70 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:44.107136 ip-10-0-128-70 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:44.107136 ip-10-0-128-70 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:44.107136 ip-10-0-128-70 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:44.107136 ip-10-0-128-70 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:44.107136 ip-10-0-128-70 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:44.107953 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.107883    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:44.112028 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112009    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.112028 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112026    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.112028 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112031    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.112028 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112035    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112039    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112058    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112062    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112065    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112069    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112073    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112077    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112081    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112085    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112088    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112093    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112099    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112105    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112110    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112114    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112118    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112129    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112133    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.112282 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112137    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112141    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112145    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112149    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112152    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112156    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112161    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112165    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112169    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112173    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112177    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112181    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112186    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112190    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112194    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112199    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112204    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112208    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112212    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112217    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.113063 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112221    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112225    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112229    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112233    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112238    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112242    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112247    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112251    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112255    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112259    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112264    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112269    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112273    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112278    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112281    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112286    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112290    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112294    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112298    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.113892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112305    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112312    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112317    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112322    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112327    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112332    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112337    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112342    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112347    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112351    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112361    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112365    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112369    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112373    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112377    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112384    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112389    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112393    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112397    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112402    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.114717 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112406    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112410    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112414    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112418    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112423    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112427    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112431    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112435    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112439    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112443    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112447    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112452    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112458    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112464    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112469    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112474    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112478    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.112483    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113648    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113659    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.115572 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113665    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113669    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113674    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113684    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113688    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113695    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113699    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113704    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113708    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113713    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113717    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113721    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113725    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113730    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113734    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113738    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113742    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113746    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113750    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.116478 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113755    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113759    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113765    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113773    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113779    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113784    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113790    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113794    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113799    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113803    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113807    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113811    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113816    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113821    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113825    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113829    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113834    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113838    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113845    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113850    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.117135 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113854    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113858    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113862    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113867    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113871    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113875    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113879    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113884    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113888    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113892    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113896    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113899    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113903    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113907    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113911    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113915    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113919    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113923    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113927    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113931    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.117643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113936    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113939    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113944    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113949    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113954    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113959    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113963    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113967    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113971    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113975    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.113997    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114003    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114007    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114011    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114015    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114020    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114024    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114028    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114032    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114036    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.118356 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114055    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114059    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114063    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114067    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114071    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114075    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114079    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114084    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114088    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114093    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114097    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114101    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114105    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114109    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114115    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114121    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114126    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114131    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114688    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.118844 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114697    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114701    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114705    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114710    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114715    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114719    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114723    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114726    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114730    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114734    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114738    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114743    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114747    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114751    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114755    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114759    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114763    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114768    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114772    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114776    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.119401 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114781    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114784    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114789    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114793    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114797    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114802    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114808    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114812    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114816    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114820    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114824    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114829    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114833    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114837    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114841    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114845    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114850    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114854    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114858    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.119894 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114862    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114866    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114869    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114875    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114879    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114883    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114887    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114891    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114895    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114899    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114903    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114907    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114911    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114916    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114920    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114924    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114928    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114932    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114936    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.120369 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114940    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114944    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114949    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114953    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114957    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114961    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114968    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114974    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114978    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114982    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114986    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114991    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114995    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.114999    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115003    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115007    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115011    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115015    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115019    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115023    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.120816 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115027    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115031    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115035    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115039    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115058    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115064    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115068    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115072    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115077    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115081    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115085    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115089    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115095    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115101    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115105    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115111    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115116    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115120    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115124    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.115128    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.121311 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115231    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115281    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115291    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115298    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115305    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115310    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115316    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115322    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115327    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115332    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115337    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115343    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115348    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115353    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115357    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115362    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115367    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115372    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115377    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115382    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115387    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115393    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115398    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115405    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:44.121888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115410    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115414    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115420    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115425    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115430    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115435    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115440    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115444    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115450    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115455    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115460    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115465    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115469    2559 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115475    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115485    2559 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115490    2559 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115495    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115500    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115504    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115510    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115515    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115520    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115525    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115530    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115534    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:44.122476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115539    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115544    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115549    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115553    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115558    2559 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115564    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115568    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115573    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115578    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115582    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115587    2559 flags.go:64] FLAG: --help="false"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115591    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115597    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115601    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115606    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115611    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115617    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115622    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115627    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115631    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115636    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115641    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115647    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115652    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:44.123168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115657    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115661    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115666    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115670    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115675    2559 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115679    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115684    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115689    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115697    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115702    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115708    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115712    2559 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115717    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115722    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115727    2559 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115731    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115738    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115743    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115749    2559 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115754    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115758    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115763    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115767    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115772    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115777    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:44.123756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115782    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115792    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115797    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115801    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115806    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115812    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115820    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115825    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115829    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115834    2559 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115839    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115844    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01129414f2d6c1b95"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115848    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115856    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115861    2559 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115866    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115871    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115876    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115880    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115885    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115891    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115896    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115902    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115906    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115911    2559 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:44.124379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115916    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115921    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115926    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115930    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115934    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115940    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115945    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115950    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115954    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115959    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115963    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115968    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115972    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115978    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115983    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115992    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.115996    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116001    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116007    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116011    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116018    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116023    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116028    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116032    2559 flags.go:64] FLAG: --v="2"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116038    2559 flags.go:64] FLAG: --version="false"
Sep 17 23:10:44.124976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116063    2559 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116069    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116076    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116239    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116248    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116255    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116260    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116264    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116269    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116273    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116277    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116282    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116286    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116290    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116295    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116299    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116303    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116307    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116311    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116315    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.125628 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116319    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116324    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116329    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116333    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116338    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116342    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116346    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116352    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116356    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116360    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116365    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116369    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116374    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116378    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116384    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116388    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116392    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116397    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116401    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116405    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.126124 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116409    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116413    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116417    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116421    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116426    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116430    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116434    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116438    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116444    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116450    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116455    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116459    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116464    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116468    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116473    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116477    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116481    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116485    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116489    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.126610 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116495    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116499    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116503    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116508    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116512    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116516    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116521    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116527    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116531    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116536    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116540    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116544    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116548    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116552    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116556    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116560    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116564    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116569    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116573    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116577    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.127102 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116581    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116585    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116589    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116593    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116597    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116602    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116607    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116611    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116616    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116620    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116624    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116630    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116637    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116641    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116645    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116650    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116654    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116659    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116665    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.127583 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116674    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116679    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116684    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.116688    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.116697    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123086    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123100    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123119    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123128    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123179    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123184    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123188    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123191    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123194    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123197    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.128059 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123200    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123202    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123205    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123208    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123211    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123214    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123216    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123219    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123222    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123225    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123227    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123230    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123232    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123236    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123240    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123243    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123246    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123249    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123252    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.128444 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123255    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123258    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123261    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123263    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123268    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123270    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123273    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123276    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123278    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123281    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123283    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123286    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123289    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123291    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123294    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123296    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123299    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123302    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123304    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123307    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.128932 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123310    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123313    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123317    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123320    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123323    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123326    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123328    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123331    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123334    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123336    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123339    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123341    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123344    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123346    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123349    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123351    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123355    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123358    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123361    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.129441 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123363    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123365    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123368    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123371    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123373    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123376    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123378    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123381    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123384    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123386    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123389    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123391    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123394    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123396    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123399    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123401    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123404    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123406    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123409    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123412    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.129889 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123414    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123417    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123419    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123422    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123424    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123427    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123429    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123432    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123436    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123440    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123444    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123447    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123450    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123452    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123455    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.130446 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123460    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123565    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123570    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123572    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123575    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123578    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123581    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123584    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123587    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123590    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123593    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123595    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123598    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123600    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123603    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123606    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123608    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123611    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123613    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123616    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:44.130811 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123618    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123621    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123623    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123626    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123628    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123631    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123634    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123637    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123639    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123642    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123645    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123647    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123650    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123652    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123655    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123658    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123660    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123663    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123665    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123668    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:44.131408 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123670    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123673    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123675    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123678    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123680    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123683    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123685    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123688    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123691    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123693    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123697    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123701    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123704    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123707    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123710    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123712    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123715    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123718    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123721    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:44.132174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123724    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123727    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123729    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123732    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123736    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123739    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123743    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123745    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123748    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123751    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123753    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123755    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123758    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123761    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123764    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123766    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123769    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123771    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123774    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:44.132731 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123776    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123779    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123782    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123784    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123787    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123790    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123792    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123795    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123799    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123802    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123807    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123810    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123813    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123816    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123819    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123822    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123824    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123828    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123831    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:44.133254 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123834    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123836    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:44.123839    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123844    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.123928    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.125733    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.126414    2559 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.126510    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.126547    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.129321    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:44.133700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.131615    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:44.142327 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.142270    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:44.144589 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.144573    2559 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:44.145494 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.145477    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:44.150216 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.150198    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 97212cf5-f54f-402c-a511-40d3d399496a:/dev/nvme0n1p4 de1aebb8-c9f3-426b-8cbc-6a163f4fcf1c:/dev/nvme0n1p3]
Sep 17 23:10:44.150282 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.150216    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:44.155118 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.154964    2559 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:44.153718255 +0000 UTC m=+0.302444580 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097064 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0dec4a68fa3b462bd3aaa99e9326 SystemUUID:ec2c0dec-4a68-fa3b-462b-d3aaa99e9326 BootID:a9a83832-e559-44f0-9289-c477a907e7b7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:2c:0b:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:2c:0b:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:2e:86:c4:cb:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:44.155118 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.155108    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:44.155234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.155178    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:44.157430 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.157407    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:44.157554 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.157432    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:44.158008 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.157999    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:44.158037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.158011    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:44.158037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.158019    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:44.158570 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.158560    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:44.158633 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.158625    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:44.161405 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161390    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:44.161511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161500    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:44.161630 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161621    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:44.161663 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161650    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:44.161663 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161659    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:44.161750 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161674    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:44.161750 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161683    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:44.161750 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.161733    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:44.162072 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.162024    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:44.162072 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.162053    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:44.163853 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.163838    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:44.166175 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.166161    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:44.166248 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.166191    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:44.167549 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167536    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:44.167595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167555    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:44.167595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167572    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:44.167595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167578    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:44.167595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167586    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:44.167595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167591    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167604    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167609    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167615    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167620    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167632    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:44.167719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.167636    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:44.168287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.168275    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:44.168323 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.168291    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:44.171735 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.171717    2559 server.go:1268] "Started kubelet"
Sep 17 23:10:44.172115 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.172061    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:44.172202 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.172181    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:44.172312 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.172151    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:44.172650 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.172615    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:44.172932 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.172882    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:44.173160 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.173128    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:44.173263 ip-10-0-128-70 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:44.174351 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.173891    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:44.177278 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.177263    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:44.179820 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.179799    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:44.180691 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.180651    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:44.180773 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.179947    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-70.ec2.internal.18d63e854d1d5f2c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-70.ec2.internal,UID:ip-10-0-128-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-70.ec2.internal,},FirstTimestamp:2026-09-17 23:10:44.171693868 +0000 UTC m=+0.320420192,LastTimestamp:2026-09-17 23:10:44.171693868 +0000 UTC m=+0.320420192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-70.ec2.internal,}"
Sep 17 23:10:44.181095 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181079    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:44.181179 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.181147    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.181561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181541    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:44.181661 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181645    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:44.181718 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181674    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:44.181823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181805    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:44.181823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181817    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:44.181984 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181970    2559 factory.go:55] Registering systemd factory
Sep 17 23:10:44.181984 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.181985    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:44.182208 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182195    2559 factory.go:153] Registering CRI-O factory
Sep 17 23:10:44.182249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182210    2559 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:44.182249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182247    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:44.182325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182259    2559 factory.go:103] Registering Raw factory
Sep 17 23:10:44.182325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182269    2559 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:44.182623 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.182610    2559 manager.go:349] Starting recovery of all containers
Sep 17 23:10:44.189119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.189092    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:44.192333 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.192307    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:44.192452 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.192427    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:44.193335 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.193313    2559 manager.go:354] Recovery completed
Sep 17 23:10:44.197992 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.197978    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.200762 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.200745    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.200824 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.200779    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.200824 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.200794    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.201400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.201385    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:44.201400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.201398    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.201508 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.201399    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:44.201508 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.201463    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:44.202080 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202066    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.202163 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202089    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.202163 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202103    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.202393 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202376    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2gxb"
Sep 17 23:10:44.202694 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202682    2559 policy_none.go:49] "None policy: Start"
Sep 17 23:10:44.202732 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202699    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:44.202732 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.202709    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:44.203752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.203740    2559 policy_none.go:47] "Start"
Sep 17 23:10:44.211882 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.211806    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-70.ec2.internal.18d63e854ed8edfb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-70.ec2.internal,UID:ip-10-0-128-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-70.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-70.ec2.internal,},FirstTimestamp:2026-09-17 23:10:44.200762875 +0000 UTC m=+0.349489204,LastTimestamp:2026-09-17 23:10:44.200762875 +0000 UTC m=+0.349489204,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-70.ec2.internal,}"
Sep 17 23:10:44.215522 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.215505    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2gxb"
Sep 17 23:10:44.230644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.230629    2559 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.230676    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.230689    2559 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.230884    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.230893    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.231000    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.231127    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.231137    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.233005    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:44.246186 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.233065    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.302183 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.302162    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:44.302256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.302204    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:44.302256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.302221    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:44.302337 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.302285    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:10:44.304392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.304375    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:44.331496 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.331468    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.332244 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.332230    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.332308 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.332255    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.332308 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.332265    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.332308 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.332284    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.339506 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.339493    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.339560 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.339511    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-70.ec2.internal\": node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.366223 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.366095    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.402896 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.402873    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"]
Sep 17 23:10:44.402973 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.402934    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.404224 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.404209    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.404303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.404238    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.404303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.404248    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.405454 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.405442    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.405568 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.405554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.405602 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.405591    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.406095 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406081    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.406154 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406104    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.406154 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406106    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.406154 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406113    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.406154 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406125    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.406154 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.406134    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.407303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.407287    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.407377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.407317    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:44.407971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.407947    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:44.407971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.407971    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:44.408088 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.407983    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:44.428626 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.428605    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-70.ec2.internal\" not found" node="ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.432830 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.432814    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-70.ec2.internal\" not found" node="ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.466470 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.466453    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.484196 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.484175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.484284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.484199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e525bd7c723bc2bc2965a954409bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"2e525bd7c723bc2bc2965a954409bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.484284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.484216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.567124 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.567108    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.584285 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.584357 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.584357 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e525bd7c723bc2bc2965a954409bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"2e525bd7c723bc2bc2965a954409bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.584445 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e525bd7c723bc2bc2965a954409bdf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"2e525bd7c723bc2bc2965a954409bdf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.584445 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584377    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.584445 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.584380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/af0a391dcf69515e2b905e618974d385-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"af0a391dcf69515e2b905e618974d385\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.667740 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.667683    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.732129 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.732111    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.737293 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:44.737280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:44.768128 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.768105    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.868590 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.868563    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:44.969056 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:44.968996    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.410619 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:46.410593    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.411963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.411951    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:46.467339 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.467322    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:46.467843 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.467433    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:46.467843 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.467466    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:46.510853 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:46.510830    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.560437 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.560375    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:45 +0000 UTC" deadline="2028-06-06 13:30:19.6720096 +0000 UTC"
Sep 17 23:10:46.560437 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.560404    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15062h19m33.111608751s"
Sep 17 23:10:46.594700 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:46.594676    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e525bd7c723bc2bc2965a954409bdf5.slice/crio-ad1c0998c1e87579de283a8e465119b8e55267de08b3a0cd0fa119d9f0d05305 WatchSource:0}: Error finding container ad1c0998c1e87579de283a8e465119b8e55267de08b3a0cd0fa119d9f0d05305: Status 404 returned error can't find the container with id ad1c0998c1e87579de283a8e465119b8e55267de08b3a0cd0fa119d9f0d05305
Sep 17 23:10:46.595241 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:46.595223    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0a391dcf69515e2b905e618974d385.slice/crio-72f25243fcbfc3683f880b5c0e9c0d9f5a8476560b8a6645c42df81a27d7ab97 WatchSource:0}: Error finding container 72f25243fcbfc3683f880b5c0e9c0d9f5a8476560b8a6645c42df81a27d7ab97: Status 404 returned error can't find the container with id 72f25243fcbfc3683f880b5c0e9c0d9f5a8476560b8a6645c42df81a27d7ab97
Sep 17 23:10:46.599810 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.599796    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:46.611712 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:46.611695    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.646264 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.646212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"af0a391dcf69515e2b905e618974d385","Type":"ContainerStarted","Data":"72f25243fcbfc3683f880b5c0e9c0d9f5a8476560b8a6645c42df81a27d7ab97"}
Sep 17 23:10:46.647285 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.647260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" event={"ID":"2e525bd7c723bc2bc2965a954409bdf5","Type":"ContainerStarted","Data":"ad1c0998c1e87579de283a8e465119b8e55267de08b3a0cd0fa119d9f0d05305"}
Sep 17 23:10:46.712442 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:46.712409    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.806239 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.806210    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:46.813345 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:46.813288    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found"
Sep 17 23:10:46.825356 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.825336    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:46.922644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.922623    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:46.944752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.944730    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:46.945630 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.945619    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"
Sep 17 23:10:46.961535 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:46.961514    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:47.502920 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.502889    2559 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:47.511490 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.511468    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:47.511867 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.511843    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9lhnq","openshift-multus/multus-smjgv","openshift-multus/network-metrics-daemon-zmndj","openshift-network-diagnostics/network-check-target-9pjl4","kube-system/konnectivity-agent-dfdfr","kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal","openshift-network-operator/iptables-alerter-5mswh","openshift-ovn-kubernetes/ovnkube-node-r92zk","kube-system/global-pull-secret-syncer-f2rvg","openshift-cluster-node-tuning-operator/tuned-t9n76","openshift-dns/node-resolver-7vljg","openshift-image-registry/node-ca-9xqrv"]
Sep 17 23:10:47.513605 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.513582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.514711 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.514688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.516123 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.516104    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.516218 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.516173    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:47.518443 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.518428    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:47.518532 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.518511    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:47.521104 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.520875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.521936 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.521910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.522342 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.522325    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:47.523080 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.523059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.524404 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.524387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.525821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.525803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.527449 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.527432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.529676 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.529640    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:47.529796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.529688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.529796 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.529740    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:47.530058 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530038    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.530193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.530296 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530234    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.530296 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530244    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gpqtf\""
Sep 17 23:10:47.530296 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:47.531007 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mhmrf\""
Sep 17 23:10:47.531086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.530991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.532998 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.532943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a85598d9-fd95-44dd-b267-ef1921ce3de0-konnectivity-ca\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.532998 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.532977    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-iptables-alerter-script\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.533149 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533009    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533510    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533762    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533789    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533938    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.533008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-socket-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-registration-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-sys-fs\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-hosts-file\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-tmp-dir\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km9m\" (UniqueName: \"kubernetes.io/projected/21037b83-b73d-4375-9b84-b018549bbc05-kube-api-access-2km9m\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-host-slash\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrdr\" (UniqueName: \"kubernetes.io/projected/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-kube-api-access-5vrdr\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-device-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a85598d9-fd95-44dd-b267-ef1921ce3de0-agent-certs\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.535526 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt6x\" (UniqueName: \"kubernetes.io/projected/44652509-993d-456f-992e-8139ee203f35-kube-api-access-mlt6x\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/449a9f0f-970a-48c3-964d-408d9328098a-host\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/449a9f0f-970a-48c3-964d-408d9328098a-serviceca\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45zw\" (UniqueName: \"kubernetes.io/projected/449a9f0f-970a-48c3-964d-408d9328098a-kube-api-access-d45zw\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534847    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534846    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx24\" (UniqueName: \"kubernetes.io/projected/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-kube-api-access-hhx24\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.534360    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rp98k\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535636    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535903    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.535955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.536191    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gqxz4\""
Sep 17 23:10:47.536435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.536250    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pwbx9\""
Sep 17 23:10:47.537063 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.536807    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zchs9\""
Sep 17 23:10:47.537063 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537035    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s2gt5\""
Sep 17 23:10:47.537149 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.537219 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537198    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.537406 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:47.537741 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:47.537821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537789    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.537891 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.537731    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.538594 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.538574    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.538730 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.538703    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8shpk\""
Sep 17 23:10:47.539290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.539266    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:47.542279 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.542259    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:47.552640 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.552621    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:47.552912 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.552894    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gsvdk\""
Sep 17 23:10:47.568943 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.568921    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:47.617393 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.617369    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkkk6"
Sep 17 23:10:47.623100 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.623079    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:47.632354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.632334    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkkk6"
Sep 17 23:10:47.636144 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a85598d9-fd95-44dd-b267-ef1921ce3de0-agent-certs\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.636241 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-system-cni-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.636241 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-cnibin\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.636327 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-netns\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636327 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-kubelet\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.636327 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-var-lib-kubelet\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-etc-tuned\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244fv\" (UniqueName: \"kubernetes.io/projected/633cba43-1482-404e-9ae7-556d35e07bff-kube-api-access-244fv\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-cnibin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljddw\" (UniqueName: \"kubernetes.io/projected/99bde14c-570c-4e9c-9224-62729c4a6ebd-kube-api-access-ljddw\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636440    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-conf\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.636497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a85598d9-fd95-44dd-b267-ef1921ce3de0-konnectivity-ca\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-registration-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-cni-binary-copy\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-conf-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovn-node-metrics-cert\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636694    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt6x\" (UniqueName: \"kubernetes.io/projected/44652509-993d-456f-992e-8139ee203f35-kube-api-access-mlt6x\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-registration-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-bin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-var-lib-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.636823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysconfig\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-sys\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-multus\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-kubelet\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-multus-certs\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-netns\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-lib-modules\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a85598d9-fd95-44dd-b267-ef1921ce3de0-konnectivity-ca\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.636983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-host\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-modprobe-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-kubernetes\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.636831    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-iptables-alerter-script\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.637104    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:48.137083035 +0000 UTC m=+2.945194173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-dbus\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.637324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-host-slash\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-host-slash\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-socket-dir-parent\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-etc-kubernetes\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-netd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-config\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-tmp\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmkd\" (UniqueName: \"kubernetes.io/projected/697bafa8-05d3-48dc-809c-e37e84377208-kube-api-access-pdmkd\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637589    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-iptables-alerter-script\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-systemd-units\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-hostroot\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-daemon-config\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-systemd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-env-overrides\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637823    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-socket-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-sys-fs\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-hosts-file\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-sys-fs\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km9m\" (UniqueName: \"kubernetes.io/projected/21037b83-b73d-4375-9b84-b018549bbc05-kube-api-access-2km9m\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.637981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrdr\" (UniqueName: \"kubernetes.io/projected/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-kube-api-access-5vrdr\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-device-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-socket-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-hosts-file\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-log-socket\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-bin\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-script-lib\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/449a9f0f-970a-48c3-964d-408d9328098a-serviceca\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.638622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638254    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-k8s-cni-cncf-io\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-ovn\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-device-dir\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-node-log\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-system-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-slash\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lf2\" (UniqueName: \"kubernetes.io/projected/0f72f03c-3028-4e21-b64e-0c2fe24e4156-kube-api-access-v9lf2\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-systemd\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx24\" (UniqueName: \"kubernetes.io/projected/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-kube-api-access-hhx24\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-tmp-dir\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-etc-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638590    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/449a9f0f-970a-48c3-964d-408d9328098a-serviceca\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/449a9f0f-970a-48c3-964d-408d9328098a-host\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45zw\" (UniqueName: \"kubernetes.io/projected/449a9f0f-970a-48c3-964d-408d9328098a-kube-api-access-d45zw\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.639288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.639862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44652509-993d-456f-992e-8139ee203f35-etc-selinux\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.639862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/449a9f0f-970a-48c3-964d-408d9328098a-host\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.639862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-kubelet-config\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.639862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-run\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.639862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.638903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-tmp-dir\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.640672 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.640637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a85598d9-fd95-44dd-b267-ef1921ce3de0-agent-certs\") pod \"konnectivity-agent-dfdfr\" (UID: \"a85598d9-fd95-44dd-b267-ef1921ce3de0\") " pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.672566 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.672455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx24\" (UniqueName: \"kubernetes.io/projected/5b41259e-6f7f-40e7-9765-c1e01f69c2c8-kube-api-access-hhx24\") pod \"node-resolver-7vljg\" (UID: \"5b41259e-6f7f-40e7-9765-c1e01f69c2c8\") " pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.673415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.673394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt6x\" (UniqueName: \"kubernetes.io/projected/44652509-993d-456f-992e-8139ee203f35-kube-api-access-mlt6x\") pod \"aws-ebs-csi-driver-node-shrht\" (UID: \"44652509-993d-456f-992e-8139ee203f35\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.674821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.674777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45zw\" (UniqueName: \"kubernetes.io/projected/449a9f0f-970a-48c3-964d-408d9328098a-kube-api-access-d45zw\") pod \"node-ca-9xqrv\" (UID: \"449a9f0f-970a-48c3-964d-408d9328098a\") " pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.676444 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.676373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrdr\" (UniqueName: \"kubernetes.io/projected/aa135e44-b6c0-42f5-b177-fb9a96c2f35a-kube-api-access-5vrdr\") pod \"iptables-alerter-5mswh\" (UID: \"aa135e44-b6c0-42f5-b177-fb9a96c2f35a\") " pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.676623 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.676572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km9m\" (UniqueName: \"kubernetes.io/projected/21037b83-b73d-4375-9b84-b018549bbc05-kube-api-access-2km9m\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:47.677088 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.677064    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:47.678054 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.677209    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:47.678054 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.677231    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:47.678054 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.677314    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:48.177295897 +0000 UTC m=+2.985407045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:47.739506 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-kubelet-config\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-run\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-system-cni-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-cnibin\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-netns\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.739627 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-kubelet-config\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-run\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-system-cni-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-cnibin\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-netns\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-kubelet\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-kubelet\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-var-lib-kubelet\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-etc-tuned\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-var-lib-kubelet\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244fv\" (UniqueName: \"kubernetes.io/projected/633cba43-1482-404e-9ae7-556d35e07bff-kube-api-access-244fv\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-cnibin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljddw\" (UniqueName: \"kubernetes.io/projected/99bde14c-570c-4e9c-9224-62729c4a6ebd-kube-api-access-ljddw\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-conf\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-cnibin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-cni-binary-copy\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-conf-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovn-node-metrics-cert\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-conf-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.739986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-conf\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-bin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-var-lib-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysconfig\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-bin\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-var-lib-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysconfig\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-sys\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-multus\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-sys\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-kubelet\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.740839 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740280    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-cni-multus\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-multus-certs\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-netns\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-multus-certs\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-lib-modules\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-var-lib-kubelet\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-host\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-run-netns\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-modprobe-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-kubernetes\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-host\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740427    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-dbus\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-lib-modules\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-socket-dir-parent\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-modprobe-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-kubernetes\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.741679 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-cni-binary-copy\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-etc-kubernetes\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-sysctl-d\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-netd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740566    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-socket-dir-parent\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b7419d9-a707-42b5-b957-000072e9b3c2-dbus\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-etc-kubernetes\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-netd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-config\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-tmp\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmkd\" (UniqueName: \"kubernetes.io/projected/697bafa8-05d3-48dc-809c-e37e84377208-kube-api-access-pdmkd\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-systemd-units\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-hostroot\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-systemd-units\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.740923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-daemon-config\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-systemd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.743315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-systemd\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-env-overrides\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-hostroot\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-config\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/697bafa8-05d3-48dc-809c-e37e84377208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-log-socket\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.741401    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-env-overrides\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-bin\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-script-lib\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:47.741465    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:48.241446231 +0000 UTC m=+3.049557381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bafa8-05d3-48dc-809c-e37e84377208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-k8s-cni-cncf-io\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-ovn\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744143 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-node-log\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-system-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-slash\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lf2\" (UniqueName: \"kubernetes.io/projected/0f72f03c-3028-4e21-b64e-0c2fe24e4156-kube-api-access-v9lf2\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-run-ovn\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-systemd\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-etc-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-log-socket\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/99bde14c-570c-4e9c-9224-62729c4a6ebd-multus-daemon-config\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-node-log\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-slash\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-system-cni-dir\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/99bde14c-570c-4e9c-9224-62729c4a6ebd-host-run-k8s-cni-cncf-io\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/633cba43-1482-404e-9ae7-556d35e07bff-etc-systemd\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741944    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovnkube-script-lib\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.744948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-etc-openvswitch\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.745794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.741977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f72f03c-3028-4e21-b64e-0c2fe24e4156-host-cni-bin\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.745794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.742221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-etc-tuned\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.745794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.742741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f72f03c-3028-4e21-b64e-0c2fe24e4156-ovn-node-metrics-cert\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.745794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.742870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/633cba43-1482-404e-9ae7-556d35e07bff-tmp\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.765626 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.765578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-244fv\" (UniqueName: \"kubernetes.io/projected/633cba43-1482-404e-9ae7-556d35e07bff-kube-api-access-244fv\") pod \"tuned-t9n76\" (UID: \"633cba43-1482-404e-9ae7-556d35e07bff\") " pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:47.767349 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.767331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmkd\" (UniqueName: \"kubernetes.io/projected/697bafa8-05d3-48dc-809c-e37e84377208-kube-api-access-pdmkd\") pod \"multus-additional-cni-plugins-9lhnq\" (UID: \"697bafa8-05d3-48dc-809c-e37e84377208\") " pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.775838 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.775806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lf2\" (UniqueName: \"kubernetes.io/projected/0f72f03c-3028-4e21-b64e-0c2fe24e4156-kube-api-access-v9lf2\") pod \"ovnkube-node-r92zk\" (UID: \"0f72f03c-3028-4e21-b64e-0c2fe24e4156\") " pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.776138 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.776122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljddw\" (UniqueName: \"kubernetes.io/projected/99bde14c-570c-4e9c-9224-62729c4a6ebd-kube-api-access-ljddw\") pod \"multus-smjgv\" (UID: \"99bde14c-570c-4e9c-9224-62729c4a6ebd\") " pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.789251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.789230    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:47.824195 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.824174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht"
Sep 17 23:10:47.832044 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.832025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vljg"
Sep 17 23:10:47.840812 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.840791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:10:47.846890 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.846875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xqrv"
Sep 17 23:10:47.855037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.855016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5mswh"
Sep 17 23:10:47.862170 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.862151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lhnq"
Sep 17 23:10:47.868157 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.868138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smjgv"
Sep 17 23:10:47.876566 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.876551    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:10:47.880278 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:47.880263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t9n76"
Sep 17 23:10:48.143509 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.143425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:48.143636 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.143613    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:48.143700 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.143689    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.143656405 +0000 UTC m=+3.951767550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:48.243965 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.243935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:48.244121 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.243987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:48.244203 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244120    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:48.244203 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244130    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:48.244203 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244147    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:48.244203 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244156    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:48.244203 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244187    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.244167822 +0000 UTC m=+4.052278965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:48.244441 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.244207    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.244197949 +0000 UTC m=+4.052309083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:48.407269 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.407239    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449a9f0f_970a_48c3_964d_408d9328098a.slice/crio-b82ec48298eac43b9e421c78e0045923fe74fd1da27bd75571d7719f4f49f97f WatchSource:0}: Error finding container b82ec48298eac43b9e421c78e0045923fe74fd1da27bd75571d7719f4f49f97f: Status 404 returned error can't find the container with id b82ec48298eac43b9e421c78e0045923fe74fd1da27bd75571d7719f4f49f97f
Sep 17 23:10:48.407786 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.407758    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697bafa8_05d3_48dc_809c_e37e84377208.slice/crio-34764c39fac2941a68dfb8a7b2035fc0610f7b64935a3407382387a38c87ddea WatchSource:0}: Error finding container 34764c39fac2941a68dfb8a7b2035fc0610f7b64935a3407382387a38c87ddea: Status 404 returned error can't find the container with id 34764c39fac2941a68dfb8a7b2035fc0610f7b64935a3407382387a38c87ddea
Sep 17 23:10:48.408740 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.408679    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b41259e_6f7f_40e7_9765_c1e01f69c2c8.slice/crio-57174cf6d51b392678ee272c2bc22136fba0ca0b1412f1f5046ca9eb8911eae0 WatchSource:0}: Error finding container 57174cf6d51b392678ee272c2bc22136fba0ca0b1412f1f5046ca9eb8911eae0: Status 404 returned error can't find the container with id 57174cf6d51b392678ee272c2bc22136fba0ca0b1412f1f5046ca9eb8911eae0
Sep 17 23:10:48.412434 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.412416    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85598d9_fd95_44dd_b267_ef1921ce3de0.slice/crio-d23e1ee4e9efd2bebe2c115d62e18264b7a05d34e3987c7e641a26509365eb03 WatchSource:0}: Error finding container d23e1ee4e9efd2bebe2c115d62e18264b7a05d34e3987c7e641a26509365eb03: Status 404 returned error can't find the container with id d23e1ee4e9efd2bebe2c115d62e18264b7a05d34e3987c7e641a26509365eb03
Sep 17 23:10:48.414086 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.414071    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bde14c_570c_4e9c_9224_62729c4a6ebd.slice/crio-421cce180f23ea213c172177ac78e309e7389b96153dad4a3bb0f329a8e27eaf WatchSource:0}: Error finding container 421cce180f23ea213c172177ac78e309e7389b96153dad4a3bb0f329a8e27eaf: Status 404 returned error can't find the container with id 421cce180f23ea213c172177ac78e309e7389b96153dad4a3bb0f329a8e27eaf
Sep 17 23:10:48.415911 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.415241    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f72f03c_3028_4e21_b64e_0c2fe24e4156.slice/crio-0024469d9962e6010ccb922e7806adfa52804e09d5b44d0692a857e4bd60b926 WatchSource:0}: Error finding container 0024469d9962e6010ccb922e7806adfa52804e09d5b44d0692a857e4bd60b926: Status 404 returned error can't find the container with id 0024469d9962e6010ccb922e7806adfa52804e09d5b44d0692a857e4bd60b926
Sep 17 23:10:48.417007 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.416175    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa135e44_b6c0_42f5_b177_fb9a96c2f35a.slice/crio-ada7101158b9fdb3d7f4180071b5c3ad49e2cec4159f985b1a18b8840d8a376f WatchSource:0}: Error finding container ada7101158b9fdb3d7f4180071b5c3ad49e2cec4159f985b1a18b8840d8a376f: Status 404 returned error can't find the container with id ada7101158b9fdb3d7f4180071b5c3ad49e2cec4159f985b1a18b8840d8a376f
Sep 17 23:10:48.417805 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.417560    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44652509_993d_456f_992e_8139ee203f35.slice/crio-16471a53ef35f3b838a789e917f34ae800a54f14311ac65bb7765a2d4e2ffa64 WatchSource:0}: Error finding container 16471a53ef35f3b838a789e917f34ae800a54f14311ac65bb7765a2d4e2ffa64: Status 404 returned error can't find the container with id 16471a53ef35f3b838a789e917f34ae800a54f14311ac65bb7765a2d4e2ffa64
Sep 17 23:10:48.419416 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:10:48.419383    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633cba43_1482_404e_9ae7_556d35e07bff.slice/crio-c057d772353bff2f510f5c95d05280fce66cc19d5938839977050da15a15c475 WatchSource:0}: Error finding container c057d772353bff2f510f5c95d05280fce66cc19d5938839977050da15a15c475: Status 404 returned error can't find the container with id c057d772353bff2f510f5c95d05280fce66cc19d5938839977050da15a15c475
Sep 17 23:10:48.492469 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.492450    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:48.634243 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.634205    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:47 +0000 UTC" deadline="2028-02-23 20:38:11.145345123 +0000 UTC"
Sep 17 23:10:48.634243 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.634241    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12573h27m22.511106189s"
Sep 17 23:10:48.643200 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.643179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:48.643295 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:48.643265    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:48.651944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.651878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dfdfr" event={"ID":"a85598d9-fd95-44dd-b267-ef1921ce3de0","Type":"ContainerStarted","Data":"d23e1ee4e9efd2bebe2c115d62e18264b7a05d34e3987c7e641a26509365eb03"}
Sep 17 23:10:48.653002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.652978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smjgv" event={"ID":"99bde14c-570c-4e9c-9224-62729c4a6ebd","Type":"ContainerStarted","Data":"421cce180f23ea213c172177ac78e309e7389b96153dad4a3bb0f329a8e27eaf"}
Sep 17 23:10:48.654118 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.654096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"0024469d9962e6010ccb922e7806adfa52804e09d5b44d0692a857e4bd60b926"}
Sep 17 23:10:48.655262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.655242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vljg" event={"ID":"5b41259e-6f7f-40e7-9765-c1e01f69c2c8","Type":"ContainerStarted","Data":"57174cf6d51b392678ee272c2bc22136fba0ca0b1412f1f5046ca9eb8911eae0"}
Sep 17 23:10:48.656873 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.656855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerStarted","Data":"34764c39fac2941a68dfb8a7b2035fc0610f7b64935a3407382387a38c87ddea"}
Sep 17 23:10:48.657910 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.657882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xqrv" event={"ID":"449a9f0f-970a-48c3-964d-408d9328098a","Type":"ContainerStarted","Data":"b82ec48298eac43b9e421c78e0045923fe74fd1da27bd75571d7719f4f49f97f"}
Sep 17 23:10:48.660218 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.660199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"af0a391dcf69515e2b905e618974d385","Type":"ContainerStarted","Data":"387a375565d0c8974e55530acf66516c15bfab9e012bae8e6ed8700b3b5b6171"}
Sep 17 23:10:48.661614 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.661590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" event={"ID":"2e525bd7c723bc2bc2965a954409bdf5","Type":"ContainerStarted","Data":"a087c3d77d84b07ad24a92b465c5a0bde81e17888ebd0413a20468197409b1d9"}
Sep 17 23:10:48.662614 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.662596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t9n76" event={"ID":"633cba43-1482-404e-9ae7-556d35e07bff","Type":"ContainerStarted","Data":"c057d772353bff2f510f5c95d05280fce66cc19d5938839977050da15a15c475"}
Sep 17 23:10:48.663573 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.663555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht" event={"ID":"44652509-993d-456f-992e-8139ee203f35","Type":"ContainerStarted","Data":"16471a53ef35f3b838a789e917f34ae800a54f14311ac65bb7765a2d4e2ffa64"}
Sep 17 23:10:48.664643 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.664625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5mswh" event={"ID":"aa135e44-b6c0-42f5-b177-fb9a96c2f35a","Type":"ContainerStarted","Data":"ada7101158b9fdb3d7f4180071b5c3ad49e2cec4159f985b1a18b8840d8a376f"}
Sep 17 23:10:48.705102 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:48.705061    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" podStartSLOduration=2.7050482049999998 podStartE2EDuration="2.705048205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:48.704140597 +0000 UTC m=+3.512251746" watchObservedRunningTime="2026-09-17 23:10:48.705048205 +0000 UTC m=+3.513159355"
Sep 17 23:10:49.151814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.151733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:49.151986 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.151907    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.151986 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.151972    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:51.15195231 +0000 UTC m=+5.960063462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.252565 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.252524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:49.252748 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.252640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:49.252852 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.252835    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:49.252923 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.252859    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:49.252923 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.252871    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.253020 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.252931    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:51.252912577 +0000 UTC m=+6.061023736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.253334 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.253309    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.253419 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.253365    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:51.253348655 +0000 UTC m=+6.061459792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.634855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.634774    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:47 +0000 UTC" deadline="2028-05-18 14:52:03.212283272 +0000 UTC"
Sep 17 23:10:49.634855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.634815    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14607h41m13.577473498s"
Sep 17 23:10:49.643788 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.643765    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:49.643911 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.643765    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:49.643911 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.643895    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:49.644008 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:49.643937    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:49.683814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.683787    2559 generic.go:356] "Generic (PLEG): container finished" podID="af0a391dcf69515e2b905e618974d385" containerID="387a375565d0c8974e55530acf66516c15bfab9e012bae8e6ed8700b3b5b6171" exitCode=0
Sep 17 23:10:49.684642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:49.684608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"af0a391dcf69515e2b905e618974d385","Type":"ContainerDied","Data":"387a375565d0c8974e55530acf66516c15bfab9e012bae8e6ed8700b3b5b6171"}
Sep 17 23:10:50.644133 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:50.644047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:50.644696 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:50.644180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:50.694404 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:50.694370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"af0a391dcf69515e2b905e618974d385","Type":"ContainerStarted","Data":"ad8875fed97ed8760e9a3bebc86a66e29b3c153c892b2b3bf7793f71a9d2895f"}
Sep 17 23:10:51.168119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:51.167560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:51.168119 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.167709    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:51.168119 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.167779    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.167760227 +0000 UTC m=+9.975871368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:51.268819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:51.268784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:51.268997 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:51.268843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:51.269060 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269029    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:51.269060 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269049    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:51.269163 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269063    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:51.269163 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269117    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.269100177 +0000 UTC m=+10.077211309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:51.269508 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269490    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:51.269598 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.269541    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.269527111 +0000 UTC m=+10.077638258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:51.645448 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:51.645355    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:51.645448 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:51.645380    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:51.645955 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.645482    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:51.645955 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:51.645569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:52.643963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:52.643725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:52.643963 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:52.643898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:53.644128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:53.644094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:53.644559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:53.644096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:53.644559 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:53.644242    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:53.644559 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:53.644353    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:54.643173 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:54.643086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:54.643336 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:54.643216    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:55.202913 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:55.202347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:55.202913 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.202503    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:55.202913 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.202568    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:03.202549937 +0000 UTC m=+18.010661088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:55.303726 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:55.303687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:55.303726 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:55.303733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:55.303938 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.303842    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:55.303938 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.303860    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:55.303938 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.303869    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:55.303938 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.303914    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:03.303901094 +0000 UTC m=+18.112012235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:55.304246 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.304228    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:55.304287 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.304273    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:03.304262389 +0000 UTC m=+18.112373531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:55.644696 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:55.644609    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:55.644850 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.644738    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:55.645112 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:55.645090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:55.645210 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:55.645190    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:56.644054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:56.644025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:56.644480 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:56.644128    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:57.643724 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:57.643691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:57.643898 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:57.643812    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:57.643898 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:57.643850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:57.644007 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:57.643927    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:10:58.643111 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:58.643081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:10:58.643554 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:58.643192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:10:59.644100 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:59.644067    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:10:59.644539 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:59.644198    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:10:59.644539 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:10:59.644240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:10:59.644539 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:10:59.644328    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:00.643952 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:00.643920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:00.644158 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:00.644031    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:01.643757 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:01.643719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:01.643932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:01.643726    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:01.643932 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:01.643883    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:01.644047 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:01.643952    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:02.644002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:02.643928    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:02.644517 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:02.644053    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:03.267766 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:03.267737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:03.267920 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.267848    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:03.267920 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.267908    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.267893109 +0000 UTC m=+34.076004237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:03.368043 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:03.368015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:03.368188 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:03.368051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:03.368188 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368147    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:03.368188 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368160    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:03.368188 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368176    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:03.368188 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368188    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:03.368366 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368213    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.368194877 +0000 UTC m=+34.176306009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:03.368366 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.368229    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.368221116 +0000 UTC m=+34.176332257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:03.644065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:03.643994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:03.644065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:03.644014    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:03.644543 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.644108    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:03.644543 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:03.644220    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:04.643002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:04.642968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:04.643154 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:04.643098    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:05.644237 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:05.644206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:05.644648 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:05.644354    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:05.644648 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:05.644408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:05.644648 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:05.644552    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:06.644048 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.643876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:06.644166 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:06.644148    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:06.727292 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.727251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dfdfr" event={"ID":"a85598d9-fd95-44dd-b267-ef1921ce3de0","Type":"ContainerStarted","Data":"e544eb38ba203f666b73581db98ab8236e240932b54953a444639f3a2bbc0207"}
Sep 17 23:11:06.729329 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.729291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smjgv" event={"ID":"99bde14c-570c-4e9c-9224-62729c4a6ebd","Type":"ContainerStarted","Data":"ec96b9605f380bc64c66eba75e2cc2b7c178de62f55130fb6702801cf407fb37"}
Sep 17 23:11:06.733052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"73f33998dfa0c0c3b9b36d7496ece13448501856b25d9681bebed38b9d3051bf"}
Sep 17 23:11:06.733153 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"1e5da9095e4faac43d50bf67cf04b3e70db8c28d3c0a990af2019e98fd541282"}
Sep 17 23:11:06.733153 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"e0759d35a7c97fab725b820c1301f7c43fd5b66ab8c76c5e6c3ec4b0e8f99325"}
Sep 17 23:11:06.733271 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"a9619407b7be52ae54d8233b9785a4b3d63a2631de782b9617839e66511b92f0"}
Sep 17 23:11:06.733271 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"eae2b27b6d5c0b553fa1947969d134acfcfdcfc3448d28587679995e9ccbb843"}
Sep 17 23:11:06.733271 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.733222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"fbdf411b6215f1133d2892e77d13dc2ecb34de2693717aa6c6d5427e76874dc2"}
Sep 17 23:11:06.735041 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.735005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vljg" event={"ID":"5b41259e-6f7f-40e7-9765-c1e01f69c2c8","Type":"ContainerStarted","Data":"b347418d364be57fb277dc19bb5abc07f630c8ad0807426930603319ddc6b5bd"}
Sep 17 23:11:06.737129 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.737079    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="37aa9d8f3fca010bd3a0a90c6f86b8cfe76cc911bc9e8745d16e94b7aa4c699e" exitCode=0
Sep 17 23:11:06.737224 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.737150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"37aa9d8f3fca010bd3a0a90c6f86b8cfe76cc911bc9e8745d16e94b7aa4c699e"}
Sep 17 23:11:06.741812 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.741786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xqrv" event={"ID":"449a9f0f-970a-48c3-964d-408d9328098a","Type":"ContainerStarted","Data":"64a1ffa4c5b7fdce1e82dd9712198ecb5de9ed5d91a6beb44efd5a316d9017ac"}
Sep 17 23:11:06.744024 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.743999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t9n76" event={"ID":"633cba43-1482-404e-9ae7-556d35e07bff","Type":"ContainerStarted","Data":"b7ddea76a2c12011a61dba71ddab1e2453819b4c9199b3f60956f9a94eac5a56"}
Sep 17 23:11:06.745905 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.745887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht" event={"ID":"44652509-993d-456f-992e-8139ee203f35","Type":"ContainerStarted","Data":"c130275ed54031b7158bdcc4892c16e6fc3cae92b27b03c85b47b5c0f4c7c642"}
Sep 17 23:11:06.751558 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.751513    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dfdfr" podStartSLOduration=4.361407315 podStartE2EDuration="21.751497925s" totalImagesPullingTime="17.39009061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.750722328 +0000 UTC m=+21.558833469" watchObservedRunningTime="2026-09-17 23:11:06.751497925 +0000 UTC m=+21.559609075"
Sep 17 23:11:06.751738 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.751705    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" podStartSLOduration=20.751697116 podStartE2EDuration="20.751697116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:50.716458246 +0000 UTC m=+5.524569398" watchObservedRunningTime="2026-09-17 23:11:06.751697116 +0000 UTC m=+21.559808267"
Sep 17 23:11:06.806520 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.806479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7vljg" podStartSLOduration=4.377123556 podStartE2EDuration="21.806466054s" totalImagesPullingTime="17.429342498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.774911708 +0000 UTC m=+21.583022857" watchObservedRunningTime="2026-09-17 23:11:06.806466054 +0000 UTC m=+21.614577203"
Sep 17 23:11:06.859825 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.859776    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9xqrv" podStartSLOduration=4.46480125 podStartE2EDuration="21.859763285s" totalImagesPullingTime="17.394962035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.859350435 +0000 UTC m=+21.667461595" watchObservedRunningTime="2026-09-17 23:11:06.859763285 +0000 UTC m=+21.667874434"
Sep 17 23:11:06.859984 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:06.859956    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-smjgv" podStartSLOduration=4.149757173 podStartE2EDuration="21.859945576s" totalImagesPullingTime="17.710188403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.831864358 +0000 UTC m=+21.639975508" watchObservedRunningTime="2026-09-17 23:11:06.859945576 +0000 UTC m=+21.668056727"
Sep 17 23:11:07.224904 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.224886    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:07.584053 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.583897    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:07.22490063Z","UUID":"38fa85f9-1bda-469c-881f-fe66cf754a92","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:07.585819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.585799    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:07.585819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.585833    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:07.643712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.643678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:07.643845 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:07.643813    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:07.643975 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.643863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:07.644088 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:07.644066    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:07.751758 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.751723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht" event={"ID":"44652509-993d-456f-992e-8139ee203f35","Type":"ContainerStarted","Data":"ec74ba0c5eabba37a00260b8d5ae96a9136c2aa15294412833c6c2c4a85f3ae1"}
Sep 17 23:11:07.753732 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.753703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5mswh" event={"ID":"aa135e44-b6c0-42f5-b177-fb9a96c2f35a","Type":"ContainerStarted","Data":"1f059cf455e9a194c9e55bb3973adcd21ff7701b7a0cd5ad224c756e84cb1690"}
Sep 17 23:11:07.843424 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.843338    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t9n76" podStartSLOduration=5.4599807909999996 podStartE2EDuration="22.843319985s" totalImagesPullingTime="17.383339194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.884299605 +0000 UTC m=+21.692410755" watchObservedRunningTime="2026-09-17 23:11:07.843319985 +0000 UTC m=+22.651431131"
Sep 17 23:11:07.843927 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:07.843886    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5mswh" podStartSLOduration=5.425575476 podStartE2EDuration="22.843875549s" totalImagesPullingTime="17.418300073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:07.826856024 +0000 UTC m=+22.634967192" watchObservedRunningTime="2026-09-17 23:11:07.843875549 +0000 UTC m=+22.651986698"
Sep 17 23:11:08.643377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:08.643347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:08.643554 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:08.643447    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:08.758726 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:08.758700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht" event={"ID":"44652509-993d-456f-992e-8139ee203f35","Type":"ContainerStarted","Data":"6950b24d656fcc3ee11e69aaff4d01a36a758eb79bd43938d70175112addc442"}
Sep 17 23:11:08.798914 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:08.798853    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shrht" podStartSLOduration=4.732918781 podStartE2EDuration="23.798833952s" totalImagesPullingTime="19.065915171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.79704583 +0000 UTC m=+23.605156983" watchObservedRunningTime="2026-09-17 23:11:08.798833952 +0000 UTC m=+23.606945103"
Sep 17 23:11:09.643767 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:09.643715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:09.644049 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:09.643834    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:09.644049 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:09.643719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:09.644049 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:09.643942    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:09.764372 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:09.764338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"fbc1019a387e31ebc76ec79aad7718ff066d5c37adfd63c64f540b51739fc95e"}
Sep 17 23:11:10.643831 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:10.643799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:10.643983 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:10.643898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:11.485916 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.485676    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:11:11.486339 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.486221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:11:11.643481 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.643451    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:11.643600 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:11.643541    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:11.643638 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.643616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:11.643720 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:11.643703    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:11.771601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.771535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" event={"ID":"0f72f03c-3028-4e21-b64e-0c2fe24e4156","Type":"ContainerStarted","Data":"686733f1854e73a5f3b362f8090a6b95721a14ac9e7abd7c669ce6b8c7d0a43e"}
Sep 17 23:11:11.771777 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.771753    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:11.771869 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.771825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:11.773484 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.773465    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="fa8ac55873f3516f59322343ed7576476e98a32f5b5a520003fe0c8d08632496" exitCode=0
Sep 17 23:11:11.773581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.773532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"fa8ac55873f3516f59322343ed7576476e98a32f5b5a520003fe0c8d08632496"}
Sep 17 23:11:11.786911 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.786894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:11.851128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.851085    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk" podStartSLOduration=8.907010058000001 podStartE2EDuration="26.851071105s" totalImagesPullingTime="17.944061047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:11.821204339 +0000 UTC m=+26.629315493" watchObservedRunningTime="2026-09-17 23:11:11.851071105 +0000 UTC m=+26.659182249"
Sep 17 23:11:11.887461 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.887443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:11:11.888013 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:11.887996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dfdfr"
Sep 17 23:11:12.643942 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.643794    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:12.644228 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:12.644019    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:12.777446 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.777419    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="47941ea9008a4953f7cbbd254b8d3f9372a464298bc9d97d82c8aaa278a79794" exitCode=0
Sep 17 23:11:12.777534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.777503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"47941ea9008a4953f7cbbd254b8d3f9372a464298bc9d97d82c8aaa278a79794"}
Sep 17 23:11:12.777845 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.777813    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:12.792057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.792008    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:12.907872 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.907848    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2rvg"]
Sep 17 23:11:12.907995 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.907941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:12.908180 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:12.908048    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:12.908617 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.908601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9pjl4"]
Sep 17 23:11:12.908702 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.908678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:12.908767 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:12.908750    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:12.909215 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.909200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmndj"]
Sep 17 23:11:12.909290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:12.909280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:12.909362 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:12.909347    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:13.782237 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:13.782204    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="c78bc10ece13e59c610977ce9ed0d437af5c5b4aee142fb0699f500ae3652a3f" exitCode=0
Sep 17 23:11:13.782816 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:13.782279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"c78bc10ece13e59c610977ce9ed0d437af5c5b4aee142fb0699f500ae3652a3f"}
Sep 17 23:11:14.643864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:14.643776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:14.644018 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:14.643901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:14.644018 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:14.643908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:14.644018 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:14.643907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:14.644173 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:14.644016    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:14.644173 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:14.644075    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:15.153465 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:15.153402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.346737 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:15.346702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.433780 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:15.433750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.570323 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:15.570286    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.688365 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:15.688326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:16.643333 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:16.643186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:16.643333 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:16.643238    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:16.643333 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:16.643321    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:16.644014 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:16.643332    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:16.644014 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:16.643412    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:16.644014 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:16.643501    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:17.280786 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:17.280546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:17.478822 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:17.478787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:17.653367 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:17.653269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:18.357228 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.357173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:18.643793 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.643768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:18.643891 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.643878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:18.643950 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.643890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:18.643950 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:18.643891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f2rvg" podUID="2b7419d9-a707-42b5-b957-000072e9b3c2"
Sep 17 23:11:18.644047 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:18.643994    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmndj" podUID="21037b83-b73d-4375-9b84-b018549bbc05"
Sep 17 23:11:18.644104 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:18.644085    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pjl4" podUID="997de6e1-d5a5-4a6e-af1f-7492043ddbc6"
Sep 17 23:11:18.795840 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.795819    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="e263bc2f00ab4ad57fee6baed92917140693d49513c0d472524909d0eb6b4fb0" exitCode=0
Sep 17 23:11:18.796137 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.795861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"e263bc2f00ab4ad57fee6baed92917140693d49513c0d472524909d0eb6b4fb0"}
Sep 17 23:11:18.958479 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.958460    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeReady"
Sep 17 23:11:18.958603 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:18.958554    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:19.031887 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.031866    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kt4mw"]
Sep 17 23:11:19.051617 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.051593    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cfl2d"]
Sep 17 23:11:19.051763 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.051744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.055694 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.055675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxcjw\""
Sep 17 23:11:19.056050 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.056033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:19.056251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.056231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:19.074870 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.074852    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cfl2d"]
Sep 17 23:11:19.074953 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.074885    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kt4mw"]
Sep 17 23:11:19.074991 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.074957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.077239 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.077213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qctjw\""
Sep 17 23:11:19.077338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.077286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:19.077889 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.077872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:19.077938 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.077896    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:19.189734 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-cert\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.189835 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89b6a55a-477e-4e8a-93f3-30d8908ba638-config-volume\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.189835 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9422n\" (UniqueName: \"kubernetes.io/projected/89b6a55a-477e-4e8a-93f3-30d8908ba638-kube-api-access-9422n\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.189901 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bnb\" (UniqueName: \"kubernetes.io/projected/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-kube-api-access-h9bnb\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.189901 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b6a55a-477e-4e8a-93f3-30d8908ba638-tmp-dir\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.189901 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.189877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.290971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.290949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-cert\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.291026 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.290976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89b6a55a-477e-4e8a-93f3-30d8908ba638-config-volume\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.291060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9422n\" (UniqueName: \"kubernetes.io/projected/89b6a55a-477e-4e8a-93f3-30d8908ba638-kube-api-access-9422n\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.291060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:19.291131 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bnb\" (UniqueName: \"kubernetes.io/projected/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-kube-api-access-h9bnb\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.291131 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b6a55a-477e-4e8a-93f3-30d8908ba638-tmp-dir\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.291131 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.291259 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.291188    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:19.291259 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.291216    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:19.291359 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.291263    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.29124375 +0000 UTC m=+66.099354892 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:19.291359 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.291292    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.791282222 +0000 UTC m=+34.599393365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:19.291465 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b6a55a-477e-4e8a-93f3-30d8908ba638-tmp-dir\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.291569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.291554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89b6a55a-477e-4e8a-93f3-30d8908ba638-config-volume\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.295242 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.295224    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-cert\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.300739 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.300720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bnb\" (UniqueName: \"kubernetes.io/projected/6be09408-eab2-4ddc-ab4a-c55fafe7dfd1-kube-api-access-h9bnb\") pod \"ingress-canary-cfl2d\" (UID: \"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1\") " pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.301823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.301803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9422n\" (UniqueName: \"kubernetes.io/projected/89b6a55a-477e-4e8a-93f3-30d8908ba638-kube-api-access-9422n\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.382883 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.382864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cfl2d"
Sep 17 23:11:19.392102 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.392083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:19.392190 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.392124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:19.392247 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392202    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:19.392298 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392256    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret podName:2b7419d9-a707-42b5-b957-000072e9b3c2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.392244939 +0000 UTC m=+66.200356071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret") pod "global-pull-secret-syncer-f2rvg" (UID: "2b7419d9-a707-42b5-b957-000072e9b3c2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:19.392298 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392276    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:19.392298 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392296    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:19.392428 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392308    2559 projected.go:196] Error preparing data for projected volume kube-api-access-kbtld for pod openshift-network-diagnostics/network-check-target-9pjl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:19.392428 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.392359    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld podName:997de6e1-d5a5-4a6e-af1f-7492043ddbc6 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.392344702 +0000 UTC m=+66.200455845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbtld" (UniqueName: "kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld") pod "network-check-target-9pjl4" (UID: "997de6e1-d5a5-4a6e-af1f-7492043ddbc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:19.719245 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.719215    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cfl2d"]
Sep 17 23:11:19.722434 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:19.722413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be09408_eab2_4ddc_ab4a_c55fafe7dfd1.slice/crio-d47cf633b618155a57b9a3dbc8d326b45ff732c93f505cacd78bfb4f5c605c84 WatchSource:0}: Error finding container d47cf633b618155a57b9a3dbc8d326b45ff732c93f505cacd78bfb4f5c605c84: Status 404 returned error can't find the container with id d47cf633b618155a57b9a3dbc8d326b45ff732c93f505cacd78bfb4f5c605c84
Sep 17 23:11:19.796234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.796172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:19.796532 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.796352    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:19.796532 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:19.796437    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:20.796413871 +0000 UTC m=+35.604525016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:19.800915 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.800896    2559 generic.go:356] "Generic (PLEG): container finished" podID="697bafa8-05d3-48dc-809c-e37e84377208" containerID="f5717cc7f81e4e9ba920a4f68844d3b5ee6072cf8761c1b729e2ef106a0b1ab1" exitCode=0
Sep 17 23:11:19.801009 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.800966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerDied","Data":"f5717cc7f81e4e9ba920a4f68844d3b5ee6072cf8761c1b729e2ef106a0b1ab1"}
Sep 17 23:11:19.802171 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.802150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cfl2d" event={"ID":"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1","Type":"ContainerStarted","Data":"d47cf633b618155a57b9a3dbc8d326b45ff732c93f505cacd78bfb4f5c605c84"}
Sep 17 23:11:19.875469 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:19.874978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:20.643419 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.643390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:20.643617 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.643508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:20.643807 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.643517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:20.646505 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.646482    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:20.647088 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.647067    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:20.647192 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.647100    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:20.647192 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.647074    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:20.647378 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.647359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfwww\""
Sep 17 23:11:20.647527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.647511    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6c57\""
Sep 17 23:11:20.801843 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.801816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:20.802440 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:20.801928    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:20.802440 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:20.801991    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:22.801972914 +0000 UTC m=+37.610084044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:20.808477 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.808455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" event={"ID":"697bafa8-05d3-48dc-809c-e37e84377208","Type":"ContainerStarted","Data":"4ebc3f10e9a332a056a0f992d9ac14c68de0f4896c9c5fcca45b2f36f887873b"}
Sep 17 23:11:20.838354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.838302    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9lhnq" podStartSLOduration=8.732992494 podStartE2EDuration="35.838286237s" totalImagesPullingTime="27.105293743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:20.835642771 +0000 UTC m=+35.643753923" watchObservedRunningTime="2026-09-17 23:11:20.838286237 +0000 UTC m=+35.646397389"
Sep 17 23:11:20.982160 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:20.982127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:21.071351 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:21.071313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:22.814743 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:22.814710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cfl2d" event={"ID":"6be09408-eab2-4ddc-ab4a-c55fafe7dfd1","Type":"ContainerStarted","Data":"0dce87887d5a571831c2368b9bf8f0569cb770c0c78177a92761890d065abc90"}
Sep 17 23:11:22.815311 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:22.815292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:22.815398 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:22.815386    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:22.815444 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:22.815435    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:26.81542072 +0000 UTC m=+41.623531848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:22.837008 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:22.836959    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cfl2d" podStartSLOduration=1.386045847 podStartE2EDuration="3.836942979s" totalImagesPullingTime="2.450897132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:22.835890002 +0000 UTC m=+37.644001152" watchObservedRunningTime="2026-09-17 23:11:22.836942979 +0000 UTC m=+37.645054130"
Sep 17 23:11:23.953949 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:23.953914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:24.022158 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:24.022129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:26.839317 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:26.839286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:26.839637 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:26.839423    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:26.839637 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:26.839472    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:34.839458962 +0000 UTC m=+49.647570093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:34.888367 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:34.888332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:34.888735 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:34.888451    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:34.888735 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:34.888513    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:50.888496746 +0000 UTC m=+65.696607876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:38.440353 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:38.440320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:38.556605 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:38.556572    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.755343 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:39.755303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.933571 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:39.933544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:40.068759 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:40.068704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:44.794052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:44.794027    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r92zk"
Sep 17 23:11:45.058857 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:45.058778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.223030 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:45.223000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.878708 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.878676    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-drfk8"]
Sep 17 23:11:49.908507 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.908483    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:49.933433 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.933399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:11:49.933433 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.933420    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 23:11:49.934361 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.934345    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:11:49.934556 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.934542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2vjr4\""
Sep 17 23:11:49.935574 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.935559    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 23:11:49.948260 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.948235    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 23:11:49.975213 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:49.975192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-drfk8"]
Sep 17 23:11:50.084975 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.084954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67517d31-1446-41a9-92ca-b6869abee076-serving-cert\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.085065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.084985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.085065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.085016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-snapshots\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.085065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.085033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps8l\" (UniqueName: \"kubernetes.io/projected/67517d31-1446-41a9-92ca-b6869abee076-kube-api-access-gps8l\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.085224 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.085114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-tmp\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.085224 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.085152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186283 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-snapshots\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps8l\" (UniqueName: \"kubernetes.io/projected/67517d31-1446-41a9-92ca-b6869abee076-kube-api-access-gps8l\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-tmp\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67517d31-1446-41a9-92ca-b6869abee076-serving-cert\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.186907 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-tmp\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.187028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.186933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/67517d31-1446-41a9-92ca-b6869abee076-snapshots\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.187106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.187032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.187153 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.187135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67517d31-1446-41a9-92ca-b6869abee076-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.189335 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.189318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67517d31-1446-41a9-92ca-b6869abee076-serving-cert\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.203977 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.203957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps8l\" (UniqueName: \"kubernetes.io/projected/67517d31-1446-41a9-92ca-b6869abee076-kube-api-access-gps8l\") pod \"insights-operator-6f9c644dd9-drfk8\" (UID: \"67517d31-1446-41a9-92ca-b6869abee076\") " pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.216804 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.216787    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8"
Sep 17 23:11:50.303292 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.303252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.448836 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.448772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.518094 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.518074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-drfk8"]
Sep 17 23:11:50.520911 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:50.520887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67517d31_1446_41a9_92ca_b6869abee076.slice/crio-d383800fe8a53eb680a73d138d15c69dc785286254105082779253b0b52efdc1 WatchSource:0}: Error finding container d383800fe8a53eb680a73d138d15c69dc785286254105082779253b0b52efdc1: Status 404 returned error can't find the container with id d383800fe8a53eb680a73d138d15c69dc785286254105082779253b0b52efdc1
Sep 17 23:11:50.871893 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.871836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8" event={"ID":"67517d31-1446-41a9-92ca-b6869abee076","Type":"ContainerStarted","Data":"d383800fe8a53eb680a73d138d15c69dc785286254105082779253b0b52efdc1"}
Sep 17 23:11:50.891113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:50.891083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:11:50.891396 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:50.891211    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:50.891396 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:50.891261    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls podName:89b6a55a-477e-4e8a-93f3-30d8908ba638 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:22.891245741 +0000 UTC m=+97.699356873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls") pod "dns-default-kt4mw" (UID: "89b6a55a-477e-4e8a-93f3-30d8908ba638") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:51.294214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.294184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:11:51.296895 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.296874    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:51.305158 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:51.305135    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:11:51.305245 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:51.305197    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs podName:21037b83-b73d-4375-9b84-b018549bbc05 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:55.305176804 +0000 UTC m=+130.113287936 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs") pod "network-metrics-daemon-zmndj" (UID: "21037b83-b73d-4375-9b84-b018549bbc05") : secret "metrics-daemon-secret" not found
Sep 17 23:11:51.395235 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.395205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:51.395370 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.395246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:51.398185 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.398160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:51.398453 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.398437    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:51.408368 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.408344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b7419d9-a707-42b5-b957-000072e9b3c2-original-pull-secret\") pod \"global-pull-secret-syncer-f2rvg\" (UID: \"2b7419d9-a707-42b5-b957-000072e9b3c2\") " pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:51.411513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.411497    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:51.419264 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.419245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtld\" (UniqueName: \"kubernetes.io/projected/997de6e1-d5a5-4a6e-af1f-7492043ddbc6-kube-api-access-kbtld\") pod \"network-check-target-9pjl4\" (UID: \"997de6e1-d5a5-4a6e-af1f-7492043ddbc6\") " pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:51.557516 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.557450    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfwww\""
Sep 17 23:11:51.562752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.562725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f2rvg"
Sep 17 23:11:51.565361 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.565337    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:51.847588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.847522    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f2rvg"]
Sep 17 23:11:51.864014 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:51.863985    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9pjl4"]
Sep 17 23:11:52.264508 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:52.264478    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7419d9_a707_42b5_b957_000072e9b3c2.slice/crio-a1e42ac26c3d45292c0f8a0f6de6c1aafe50d4f0be7865dd0665e3f898a38b00 WatchSource:0}: Error finding container a1e42ac26c3d45292c0f8a0f6de6c1aafe50d4f0be7865dd0665e3f898a38b00: Status 404 returned error can't find the container with id a1e42ac26c3d45292c0f8a0f6de6c1aafe50d4f0be7865dd0665e3f898a38b00
Sep 17 23:11:52.265044 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:52.265022    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997de6e1_d5a5_4a6e_af1f_7492043ddbc6.slice/crio-7928312c20a7c3b72335063bf1723bdf43dc2760c6861bffc682fefd81eb9599 WatchSource:0}: Error finding container 7928312c20a7c3b72335063bf1723bdf43dc2760c6861bffc682fefd81eb9599: Status 404 returned error can't find the container with id 7928312c20a7c3b72335063bf1723bdf43dc2760c6861bffc682fefd81eb9599
Sep 17 23:11:52.879117 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:52.879049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9pjl4" event={"ID":"997de6e1-d5a5-4a6e-af1f-7492043ddbc6","Type":"ContainerStarted","Data":"7928312c20a7c3b72335063bf1723bdf43dc2760c6861bffc682fefd81eb9599"}
Sep 17 23:11:52.880733 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:52.880694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2rvg" event={"ID":"2b7419d9-a707-42b5-b957-000072e9b3c2","Type":"ContainerStarted","Data":"a1e42ac26c3d45292c0f8a0f6de6c1aafe50d4f0be7865dd0665e3f898a38b00"}
Sep 17 23:11:52.883159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:52.882772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8" event={"ID":"67517d31-1446-41a9-92ca-b6869abee076","Type":"ContainerStarted","Data":"ff30dd1b1f6f038463cab1a3f6d037e7a7c7cee3ce8d18cad1ea7894d0940d6e"}
Sep 17 23:11:52.912651 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:52.912613    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8" podStartSLOduration=2.147675095 podStartE2EDuration="3.912598481s" totalImagesPullingTime="1.764923386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:52.910952378 +0000 UTC m=+67.719063528" watchObservedRunningTime="2026-09-17 23:11:52.912598481 +0000 UTC m=+67.720709633"
Sep 17 23:11:53.069428 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:53.069395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:53.168307 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:53.168177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.117294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.117133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.149915 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.149885    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xsdpn"]
Sep 17 23:11:54.152120 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.152096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.156577 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.156556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:11:54.156577 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.156569    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:11:54.156878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.156730    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q55mc\""
Sep 17 23:11:54.173758 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.173688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xsdpn"]
Sep 17 23:11:54.256269 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.256142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.313789 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.313751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-crio-socket\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.313947 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.313832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.314013 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.313947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-data-volume\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.314013 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.313990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.314112 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.314088    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wc4\" (UniqueName: \"kubernetes.io/projected/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-api-access-56wc4\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414509 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414436    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wc4\" (UniqueName: \"kubernetes.io/projected/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-api-access-56wc4\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414509 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-crio-socket\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-data-volume\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-crio-socket\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.414727 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.414681    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.414980 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.414741    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.914726409 +0000 UTC m=+69.722837563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.414980 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.414747    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.414980 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.414842    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.914812182 +0000 UTC m=+69.722923323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.414980 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.414942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-data-volume\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.424166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.424127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wc4\" (UniqueName: \"kubernetes.io/projected/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-api-access-56wc4\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.916973 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.916943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.917142 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:54.917015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:54.917142 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.917074    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.917142 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.917130    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.917290 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.917148    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.917131641 +0000 UTC m=+70.725242776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.917290 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:54.917169    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.917157819 +0000 UTC m=+70.725268950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:55.833728 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:55.833698    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7vljg_5b41259e-6f7f-40e7-9765-c1e01f69c2c8/dns-node-resolver/0.log"
Sep 17 23:11:55.922953 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:55.922922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:55.923132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:55.922983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:55.923132 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:55.923086    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:55.923132 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:55.923097    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:55.923300 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:55.923140    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.92312426 +0000 UTC m=+72.731235408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:55.923300 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:55.923173    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.923150914 +0000 UTC m=+72.731262060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:56.635871 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.635842    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xqrv_449a9f0f-970a-48c3-964d-408d9328098a/node-ca/0.log"
Sep 17 23:11:56.895612 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.895525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f2rvg" event={"ID":"2b7419d9-a707-42b5-b957-000072e9b3c2","Type":"ContainerStarted","Data":"57cb74972c2a5e7388c35b5a03a1e17f105b1511f3214bdc5d32d1faa5c8e167"}
Sep 17 23:11:56.896959 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.896933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9pjl4" event={"ID":"997de6e1-d5a5-4a6e-af1f-7492043ddbc6","Type":"ContainerStarted","Data":"cb4a9eabcf1e1cf598fe98ae13a168273a3e98b4cfdfe90c21786c3c4c67683d"}
Sep 17 23:11:56.897089 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.897073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:11:56.913788 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.913747    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f2rvg" podStartSLOduration=67.675762718 podStartE2EDuration="1m11.91373469s" totalImagesPullingTime="4.237971972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:56.912807361 +0000 UTC m=+71.720918501" watchObservedRunningTime="2026-09-17 23:11:56.91373469 +0000 UTC m=+71.721845839"
Sep 17 23:11:56.931888 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:56.931854    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9pjl4" podStartSLOduration=67.699516004 podStartE2EDuration="1m11.93184496s" totalImagesPullingTime="4.232328956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:56.931550499 +0000 UTC m=+71.739661647" watchObservedRunningTime="2026-09-17 23:11:56.93184496 +0000 UTC m=+71.739956108"
Sep 17 23:11:57.636171 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.636140    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cfl2d_6be09408-eab2-4ddc-ab4a-c55fafe7dfd1/serve-healthcheck-canary/0.log"
Sep 17 23:11:57.856260 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.856230    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"]
Sep 17 23:11:57.859116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.859100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:57.862982 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.862961    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:11:57.863259 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.863243    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:11:57.863351 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.863303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:11:57.864337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.864321    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 23:11:57.876858 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.876838    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"]
Sep 17 23:11:57.879381 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.879367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:57.882443 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.882422    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hx9r9\""
Sep 17 23:11:57.882819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.882806    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 23:11:57.906144 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.906091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"]
Sep 17 23:11:57.935308 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-tmp\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:57.935387 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85mt\" (UniqueName: \"kubernetes.io/projected/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-kube-api-access-l85mt\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:57.935387 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:57.935504 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:57.935504 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:57.935487    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:57.935706 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:57.935552    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:01.935535295 +0000 UTC m=+76.743646441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:57.935706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a574d794-0abe-48ee-a6e2-f5c7580009c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:57.935706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmjg\" (UniqueName: \"kubernetes.io/projected/a574d794-0abe-48ee-a6e2-f5c7580009c1-kube-api-access-hhmjg\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:57.935706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.935673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:11:57.935867 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:57.935731    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:57.935867 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:11:57.935781    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:01.935771126 +0000 UTC m=+76.743882287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:57.955831 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:57.955809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"]
Sep 17 23:11:58.036473 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a574d794-0abe-48ee-a6e2-f5c7580009c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:58.036563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmjg\" (UniqueName: \"kubernetes.io/projected/a574d794-0abe-48ee-a6e2-f5c7580009c1-kube-api-access-hhmjg\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:58.036563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-tmp\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.036563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85mt\" (UniqueName: \"kubernetes.io/projected/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-kube-api-access-l85mt\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.036735 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.036916 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.036896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-tmp\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.038869 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.038851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a574d794-0abe-48ee-a6e2-f5c7580009c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:58.038933 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.038886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.053806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.053789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85mt\" (UniqueName: \"kubernetes.io/projected/c89aeb02-efaf-41ca-9843-be4ae42f2d6c-kube-api-access-l85mt\") pod \"klusterlet-addon-workmgr-59b9fd4fd4-rmbs9\" (UID: \"c89aeb02-efaf-41ca-9843-be4ae42f2d6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.054010 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.053994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmjg\" (UniqueName: \"kubernetes.io/projected/a574d794-0abe-48ee-a6e2-f5c7580009c1-kube-api-access-hhmjg\") pod \"managed-serviceaccount-addon-agent-6f6784b488-f8zvf\" (UID: \"a574d794-0abe-48ee-a6e2-f5c7580009c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:58.167847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.167804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:11:58.200376 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.200323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"
Sep 17 23:11:58.438273 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.438243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"]
Sep 17 23:11:58.440879 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:58.440858    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89aeb02_efaf_41ca_9843_be4ae42f2d6c.slice/crio-42f08c258d629b03d0d15a7e5df8f1a9ee6ff858e891fcd1ad031db13598bccd WatchSource:0}: Error finding container 42f08c258d629b03d0d15a7e5df8f1a9ee6ff858e891fcd1ad031db13598bccd: Status 404 returned error can't find the container with id 42f08c258d629b03d0d15a7e5df8f1a9ee6ff858e891fcd1ad031db13598bccd
Sep 17 23:11:58.484072 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.484043    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf"]
Sep 17 23:11:58.486648 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:11:58.486626    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda574d794_0abe_48ee_a6e2_f5c7580009c1.slice/crio-2d256d77591cbd5f936635f8c2dcce2b7e57d5c6abbb7b6046980e5858f33841 WatchSource:0}: Error finding container 2d256d77591cbd5f936635f8c2dcce2b7e57d5c6abbb7b6046980e5858f33841: Status 404 returned error can't find the container with id 2d256d77591cbd5f936635f8c2dcce2b7e57d5c6abbb7b6046980e5858f33841
Sep 17 23:11:58.902758 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.902677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9" event={"ID":"c89aeb02-efaf-41ca-9843-be4ae42f2d6c","Type":"ContainerStarted","Data":"42f08c258d629b03d0d15a7e5df8f1a9ee6ff858e891fcd1ad031db13598bccd"}
Sep 17 23:11:58.903687 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:11:58.903648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf" event={"ID":"a574d794-0abe-48ee-a6e2-f5c7580009c1","Type":"ContainerStarted","Data":"2d256d77591cbd5f936635f8c2dcce2b7e57d5c6abbb7b6046980e5858f33841"}
Sep 17 23:12:00.010099 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.010063    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"]
Sep 17 23:12:00.013911 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.013085    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.021607 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.020910    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 23:12:00.021607 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.020973    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w4gnj\""
Sep 17 23:12:00.021607 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.020918    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 23:12:00.021607 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.021270    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:00.021607 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.021460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:00.036842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.036790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"]
Sep 17 23:12:00.054002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.053973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.054124 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.054039    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jr8\" (UniqueName: \"kubernetes.io/projected/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-kube-api-access-58jr8\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.054124 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.054098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.155865 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.155832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.156001 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.155877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jr8\" (UniqueName: \"kubernetes.io/projected/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-kube-api-access-58jr8\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.156001 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.155924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.156533 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.156509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.158904 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.158884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.170866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.170823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jr8\" (UniqueName: \"kubernetes.io/projected/95e5e3d6-dbcb-4de2-a4de-a26a8c94c301-kube-api-access-58jr8\") pod \"kube-storage-version-migrator-operator-55889f7b47-gpvx7\" (UID: \"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.328188 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.328106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"
Sep 17 23:12:00.632824 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.632730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:00.772125 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:00.772010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:01.972098 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.972067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:01.972534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.972121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:01.972534 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:01.972208    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:01.972534 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:01.972253    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:01.972534 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:01.972266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:09.972253559 +0000 UTC m=+84.780364702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:01.972534 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:01.972309    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:09.972296352 +0000 UTC m=+84.780407479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:01.988324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.988294    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:12:01.991227 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.991210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:01.995249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.995228    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:12:01.995411 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.995392    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gj7c\""
Sep 17 23:12:01.995489 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.995462    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:12:01.995554 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:01.995488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:12:02.014598 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.014569    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:12:02.041608 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.041580    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:12:02.072951 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.072924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.072972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073197 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073267 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcbb\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.073375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.073314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174584 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcbb\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.174752 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.175003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.175003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.174834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.175003 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.174973    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:02.175003 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.174995    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846d65468-dz2vm: secret "image-registry-tls" not found
Sep 17 23:12:02.175195 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.175067    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls podName:1fcc5758-af5e-4ff1-b9b5-a644331433b5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:02.675046565 +0000 UTC m=+77.483157704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls") pod "image-registry-5846d65468-dz2vm" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5") : secret "image-registry-tls" not found
Sep 17 23:12:02.175294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.175261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.175393 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.175370    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.175743 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.175714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.177459 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.177435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.177545 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.177508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.197619 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.197600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcbb\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.201396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.201371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.680415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:02.680382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:02.680594 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.680544    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:02.680594 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.680565    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846d65468-dz2vm: secret "image-registry-tls" not found
Sep 17 23:12:02.680745 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:02.680637    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls podName:1fcc5758-af5e-4ff1-b9b5-a644331433b5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:03.680616421 +0000 UTC m=+78.488727556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls") pod "image-registry-5846d65468-dz2vm" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5") : secret "image-registry-tls" not found
Sep 17 23:12:03.302718 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.302628    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7"]
Sep 17 23:12:03.304701 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:03.304655    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e5e3d6_dbcb_4de2_a4de_a26a8c94c301.slice/crio-6bc61471bb5d18eba4adfc58416639db856a665f94f39c483b0e9eb37286226b WatchSource:0}: Error finding container 6bc61471bb5d18eba4adfc58416639db856a665f94f39c483b0e9eb37286226b: Status 404 returned error can't find the container with id 6bc61471bb5d18eba4adfc58416639db856a665f94f39c483b0e9eb37286226b
Sep 17 23:12:03.689373 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.689340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:03.689531 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:03.689513    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:03.689586 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:03.689534    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846d65468-dz2vm: secret "image-registry-tls" not found
Sep 17 23:12:03.689623 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:03.689596    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls podName:1fcc5758-af5e-4ff1-b9b5-a644331433b5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:05.689577672 +0000 UTC m=+80.497688801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls") pod "image-registry-5846d65468-dz2vm" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5") : secret "image-registry-tls" not found
Sep 17 23:12:03.920841 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.920798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf" event={"ID":"a574d794-0abe-48ee-a6e2-f5c7580009c1","Type":"ContainerStarted","Data":"1d2cb054bab34134cff5b4be9fe80490b23ca4dc5b841d4d5923755465112534"}
Sep 17 23:12:03.922107 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.922071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7" event={"ID":"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301","Type":"ContainerStarted","Data":"6bc61471bb5d18eba4adfc58416639db856a665f94f39c483b0e9eb37286226b"}
Sep 17 23:12:03.923788 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.923762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9" event={"ID":"c89aeb02-efaf-41ca-9843-be4ae42f2d6c","Type":"ContainerStarted","Data":"158492b2a57f9510da782d722b0571aa4092fa6b4f549cb2ffe99027d6c10b63"}
Sep 17 23:12:03.923982 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.923961    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:12:03.926002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.925986    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9"
Sep 17 23:12:03.940087 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.940006    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f6784b488-f8zvf" podStartSLOduration=2.384626107 podStartE2EDuration="6.939987351s" totalImagesPullingTime="4.555361244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:03.937901146 +0000 UTC m=+78.746012296" watchObservedRunningTime="2026-09-17 23:12:03.939987351 +0000 UTC m=+78.748098502"
Sep 17 23:12:03.961288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:03.961249    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59b9fd4fd4-rmbs9" podStartSLOduration=2.350994552 podStartE2EDuration="6.961237238s" totalImagesPullingTime="4.610242686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:03.961051313 +0000 UTC m=+78.769162464" watchObservedRunningTime="2026-09-17 23:12:03.961237238 +0000 UTC m=+78.769348388"
Sep 17 23:12:04.496933 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:04.496903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:04.576443 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:04.576406    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:04.922056 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:04.922028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:04.929927 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:04.929896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7" event={"ID":"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301","Type":"ContainerStarted","Data":"dea9ef7c87c29e09cc30761c0d98961a96ea2e61b91c4e334ca358c65e9cf3fe"}
Sep 17 23:12:04.961052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:04.961011    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7" podStartSLOduration=4.424811754 podStartE2EDuration="5.960995279s" totalImagesPullingTime="1.536183525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:04.960440398 +0000 UTC m=+79.768551552" watchObservedRunningTime="2026-09-17 23:12:04.960995279 +0000 UTC m=+79.769106463"
Sep 17 23:12:05.022896 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:05.022865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.205546 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:05.205513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.308138 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:05.308106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:05.706059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:05.706030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:05.706537 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:05.706188    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:05.706537 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:05.706208    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846d65468-dz2vm: secret "image-registry-tls" not found
Sep 17 23:12:05.706537 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:05.706262    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls podName:1fcc5758-af5e-4ff1-b9b5-a644331433b5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:09.706246861 +0000 UTC m=+84.514357988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls") pod "image-registry-5846d65468-dz2vm" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5") : secret "image-registry-tls" not found
Sep 17 23:12:06.135457 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.135380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.234015 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.233982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.342462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.342430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.792357 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.792325    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qdxdm"]
Sep 17 23:12:06.794242 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.794227    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:06.797211 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.797181    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 23:12:06.797345 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.797223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 23:12:06.797345 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.797226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 23:12:06.798126 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.798110    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 23:12:06.798993 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.798977    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6k6ns\""
Sep 17 23:12:06.820673 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.820634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qdxdm"]
Sep 17 23:12:06.880060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.880035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.916609 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.916587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-key\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:06.916695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.916623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwp2\" (UniqueName: \"kubernetes.io/projected/3dfd959e-9114-4d4a-966f-edc99248a1c6-kube-api-access-cxwp2\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:06.916741 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:06.916722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-cabundle\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.017908 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.017876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwp2\" (UniqueName: \"kubernetes.io/projected/3dfd959e-9114-4d4a-966f-edc99248a1c6-kube-api-access-cxwp2\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.017999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.017951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-cabundle\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.018042 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.018004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-key\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.018578 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.018555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-cabundle\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.020404 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.020363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3dfd959e-9114-4d4a-966f-edc99248a1c6-signing-key\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.032155 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.032129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwp2\" (UniqueName: \"kubernetes.io/projected/3dfd959e-9114-4d4a-966f-edc99248a1c6-kube-api-access-cxwp2\") pod \"service-ca-55f6fc7b86-qdxdm\" (UID: \"3dfd959e-9114-4d4a-966f-edc99248a1c6\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.037869 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.037847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:07.102891 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.102846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm"
Sep 17 23:12:07.388141 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.388077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qdxdm"]
Sep 17 23:12:07.391215 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.391185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:07.391842 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:07.391823    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfd959e_9114_4d4a_966f_edc99248a1c6.slice/crio-2be442ff23a8f53fabb7994827e4d5c843a2fb27c91f7a024eb87ebf6a0c7b13 WatchSource:0}: Error finding container 2be442ff23a8f53fabb7994827e4d5c843a2fb27c91f7a024eb87ebf6a0c7b13: Status 404 returned error can't find the container with id 2be442ff23a8f53fabb7994827e4d5c843a2fb27c91f7a024eb87ebf6a0c7b13
Sep 17 23:12:07.941008 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:07.940965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm" event={"ID":"3dfd959e-9114-4d4a-966f-edc99248a1c6","Type":"ContainerStarted","Data":"2be442ff23a8f53fabb7994827e4d5c843a2fb27c91f7a024eb87ebf6a0c7b13"}
Sep 17 23:12:09.008378 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:09.008287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:09.741102 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:09.741072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:09.741269 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:09.741203    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:09.741269 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:09.741220    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5846d65468-dz2vm: secret "image-registry-tls" not found
Sep 17 23:12:09.741269 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:09.741266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls podName:1fcc5758-af5e-4ff1-b9b5-a644331433b5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:17.74125266 +0000 UTC m=+92.549363794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls") pod "image-registry-5846d65468-dz2vm" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5") : secret "image-registry-tls" not found
Sep 17 23:12:09.949099 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:09.949063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm" event={"ID":"3dfd959e-9114-4d4a-966f-edc99248a1c6","Type":"ContainerStarted","Data":"c38b384a653a59e5c48fb19cd6b0ac11dc3eeb69a54fc7ce6876d61e6e4b393a"}
Sep 17 23:12:09.978902 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:09.978855    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-qdxdm" podStartSLOduration=2.292238733 podStartE2EDuration="3.978839565s" totalImagesPullingTime="1.686600832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:09.976995893 +0000 UTC m=+84.785107043" watchObservedRunningTime="2026-09-17 23:12:09.978839565 +0000 UTC m=+84.786950740"
Sep 17 23:12:10.043311 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:10.043245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:10.043727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:10.043309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:10.043727 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:10.043392    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:10.043727 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:10.043452    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls podName:0f5dd514-e15b-4f12-b583-e39c4e9c6bc2 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:26.043436811 +0000 UTC m=+100.851547964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xsdpn" (UID: "0f5dd514-e15b-4f12-b583-e39c4e9c6bc2") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:10.043955 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:10.043931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:10.758463 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:10.758423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:11.036632 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:11.036540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:11.176291 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:11.176260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:11.286007 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:11.285968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:11.507079 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:11.507043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:12.678216 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:12.678183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:12.873963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:12.873933    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:13.114856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:13.114819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:13.278398 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:13.278366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:13.501339 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:13.501307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:14.466308 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:14.466257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:14.564003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:14.563965    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:14.642429 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:14.642395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:14.721327 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:14.721244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:14.840391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:14.840358    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:15.127926 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:15.127848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:15.232054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:15.232019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:15.329637 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:15.329608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:15.836052 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:15.836015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.450615 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:16.450584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.669056 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:16.669027    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.971939 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:16.971910    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:17.279610 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.279529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:17.409589 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.409552    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:17.803649 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.803615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:17.806024 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.806004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"image-registry-5846d65468-dz2vm\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") " pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:17.867392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.867344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:17.901233 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:17.901209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:18.189355 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.189328    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:12:18.191643 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:18.191608    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcc5758_af5e_4ff1_b9b5_a644331433b5.slice/crio-cfed52b8a3aaedb305736397bfa40706142bbf8b325cb30c856ae0595e5f4b67 WatchSource:0}: Error finding container cfed52b8a3aaedb305736397bfa40706142bbf8b325cb30c856ae0595e5f4b67: Status 404 returned error can't find the container with id cfed52b8a3aaedb305736397bfa40706142bbf8b325cb30c856ae0595e5f4b67
Sep 17 23:12:18.438342 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.438294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.632601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.632502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.832370 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.832332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:18.978422 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.978381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" event={"ID":"1fcc5758-af5e-4ff1-b9b5-a644331433b5","Type":"ContainerStarted","Data":"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"}
Sep 17 23:12:18.978600 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.978441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" event={"ID":"1fcc5758-af5e-4ff1-b9b5-a644331433b5","Type":"ContainerStarted","Data":"cfed52b8a3aaedb305736397bfa40706142bbf8b325cb30c856ae0595e5f4b67"}
Sep 17 23:12:18.978600 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:18.978525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:19.004801 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:19.004756    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" podStartSLOduration=18.004740626 podStartE2EDuration="18.004740626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:19.00360636 +0000 UTC m=+93.811717514" watchObservedRunningTime="2026-09-17 23:12:19.004740626 +0000 UTC m=+93.812851775"
Sep 17 23:12:19.246872 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:19.246779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:19.340360 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:19.340324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:19.632260 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:19.632174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:20.402543 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:20.402506    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:20.501872 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:20.501836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:20.846113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:20.846079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:21.075856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:21.075814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:21.265450 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:21.265414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:21.558145 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:21.558055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:22.944015 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:22.943984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:12:22.946059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:22.946032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89b6a55a-477e-4e8a-93f3-30d8908ba638-metrics-tls\") pod \"dns-default-kt4mw\" (UID: \"89b6a55a-477e-4e8a-93f3-30d8908ba638\") " pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:12:22.963964 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:22.963942    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxcjw\""
Sep 17 23:12:22.971595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:22.971580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:12:23.244495 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:23.244428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kt4mw"]
Sep 17 23:12:23.247174 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:23.247150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b6a55a_477e_4e8a_93f3_30d8908ba638.slice/crio-145195b71f026ad29678d7cea91045047723196dada9f8fe289a0ad8310ea0c0 WatchSource:0}: Error finding container 145195b71f026ad29678d7cea91045047723196dada9f8fe289a0ad8310ea0c0: Status 404 returned error can't find the container with id 145195b71f026ad29678d7cea91045047723196dada9f8fe289a0ad8310ea0c0
Sep 17 23:12:23.995600 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:23.995558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt4mw" event={"ID":"89b6a55a-477e-4e8a-93f3-30d8908ba638","Type":"ContainerStarted","Data":"145195b71f026ad29678d7cea91045047723196dada9f8fe289a0ad8310ea0c0"}
Sep 17 23:12:24.801906 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:24.801877    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:24.892786 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:24.892743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:24.992926 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:24.992837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:25.002998 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:25.002970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt4mw" event={"ID":"89b6a55a-477e-4e8a-93f3-30d8908ba638","Type":"ContainerStarted","Data":"ee18dd23603096976b4ed95ce389fd36c65a48275a8cd619906ae798a2c59bbb"}
Sep 17 23:12:25.003403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:25.003026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt4mw" event={"ID":"89b6a55a-477e-4e8a-93f3-30d8908ba638","Type":"ContainerStarted","Data":"1a877a667d2decf309b7bec021c12f03b59698f201266a02667b14b9244378ab"}
Sep 17 23:12:25.003403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:25.003119    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:12:25.030800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:25.030749    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kt4mw" podStartSLOduration=64.592716477 podStartE2EDuration="1m6.030731044s" totalImagesPullingTime="1.438014567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:25.02939581 +0000 UTC m=+99.837506960" watchObservedRunningTime="2026-09-17 23:12:25.030731044 +0000 UTC m=+99.838842197"
Sep 17 23:12:25.184553 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:25.184523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:26.068383 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:26.068351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:26.070498 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:26.070479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f5dd514-e15b-4f12-b583-e39c4e9c6bc2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsdpn\" (UID: \"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2\") " pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:26.262111 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:26.262080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xsdpn"
Sep 17 23:12:26.537956 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:26.537926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xsdpn"]
Sep 17 23:12:26.540837 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:26.540801    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5dd514_e15b_4f12_b583_e39c4e9c6bc2.slice/crio-4e0a39d4d9665603fa85b34207d2d9184021997d6d94a7bac63f8728915af89e WatchSource:0}: Error finding container 4e0a39d4d9665603fa85b34207d2d9184021997d6d94a7bac63f8728915af89e: Status 404 returned error can't find the container with id 4e0a39d4d9665603fa85b34207d2d9184021997d6d94a7bac63f8728915af89e
Sep 17 23:12:27.011676 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:27.011627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsdpn" event={"ID":"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2","Type":"ContainerStarted","Data":"6bb1841bc65fb6b5ce347ecdcbe94a6b3d0a194b764b8d5c7be11b56c60a1981"}
Sep 17 23:12:27.011903 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:27.011714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsdpn" event={"ID":"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2","Type":"ContainerStarted","Data":"4e0a39d4d9665603fa85b34207d2d9184021997d6d94a7bac63f8728915af89e"}
Sep 17 23:12:27.901644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:27.901573    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9pjl4"
Sep 17 23:12:28.016558 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:28.016520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsdpn" event={"ID":"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2","Type":"ContainerStarted","Data":"cb0e7799eb12545aa73441ab50c19716bb0b36e9699be4925cd1dff9f81de96d"}
Sep 17 23:12:30.024608 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:30.024562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsdpn" event={"ID":"0f5dd514-e15b-4f12-b583-e39c4e9c6bc2","Type":"ContainerStarted","Data":"9ba7ae4b56251a81ffe62103ed17486b678e3a43e214c66303790f2df532f09e"}
Sep 17 23:12:30.441835 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:30.441800    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:30.553111 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:30.553076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:35.008834 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.008801    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kt4mw"
Sep 17 23:12:35.034926 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.034876    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xsdpn" podStartSLOduration=38.488524351 podStartE2EDuration="41.034861857s" totalImagesPullingTime="2.546337506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:30.366802611 +0000 UTC m=+105.174913760" watchObservedRunningTime="2026-09-17 23:12:35.034861857 +0000 UTC m=+109.842973006"
Sep 17 23:12:35.145604 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.145576    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:12:35.149617 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.149589    2559 patch_prober.go:28] interesting pod/image-registry-5846d65468-dz2vm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:12:35.149756 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.149636    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:12:35.239675 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.239630    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"]
Sep 17 23:12:35.294170 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.294106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:35.297288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.297271    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"]
Sep 17 23:12:35.297350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.297345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:35.300399 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.300379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 23:12:35.300508 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.300380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p8llf\""
Sep 17 23:12:35.420229 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.420206    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:35.437182 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.437159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/508ac08c-a3ce-40f6-82e5-775f03337d75-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-wb7lm\" (UID: \"508ac08c-a3ce-40f6-82e5-775f03337d75\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:35.537954 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.537924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/508ac08c-a3ce-40f6-82e5-775f03337d75-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-wb7lm\" (UID: \"508ac08c-a3ce-40f6-82e5-775f03337d75\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:35.540210 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.540186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/508ac08c-a3ce-40f6-82e5-775f03337d75-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-wb7lm\" (UID: \"508ac08c-a3ce-40f6-82e5-775f03337d75\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:35.606220 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.606165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:35.911285 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:35.911210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"]
Sep 17 23:12:35.913821 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:35.913796    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ac08c_a3ce_40f6_82e5_775f03337d75.slice/crio-de0f3d6d161999afa0ea6f78deb1b4603854bfe9882446d7a5e09ae71c278ff4 WatchSource:0}: Error finding container de0f3d6d161999afa0ea6f78deb1b4603854bfe9882446d7a5e09ae71c278ff4: Status 404 returned error can't find the container with id de0f3d6d161999afa0ea6f78deb1b4603854bfe9882446d7a5e09ae71c278ff4
Sep 17 23:12:36.043164 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:36.043126    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm" event={"ID":"508ac08c-a3ce-40f6-82e5-775f03337d75","Type":"ContainerStarted","Data":"de0f3d6d161999afa0ea6f78deb1b4603854bfe9882446d7a5e09ae71c278ff4"}
Sep 17 23:12:37.050051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.050024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm" event={"ID":"508ac08c-a3ce-40f6-82e5-775f03337d75","Type":"ContainerStarted","Data":"20e6307491d484432f1de434116bbbcaf0d676f2f1448b6f5d8fc367b3123201"}
Sep 17 23:12:37.050433 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.050231    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:37.054845 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.054826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm"
Sep 17 23:12:37.110508 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.110468    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-wb7lm" podStartSLOduration=1.055434959 podStartE2EDuration="2.110454355s" totalImagesPullingTime="1.055019396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:37.080378509 +0000 UTC m=+111.888489659" watchObservedRunningTime="2026-09-17 23:12:37.110454355 +0000 UTC m=+111.918565504"
Sep 17 23:12:37.679323 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.679293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:37.764367 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:37.764337    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.052523 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:38.052427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.166403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:38.166375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.635271 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:38.635244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.879108 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:38.879078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.989971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:38.989942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.161340 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:39.161313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.245559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:39.245498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.786036 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:40.786002    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.423724 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.423686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.604457 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.604423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.791978 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.791941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.822571 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.822539    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"]
Sep 17 23:12:42.825041 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.825024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.827695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.827671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:12:42.827695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.827688    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 23:12:42.828109 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.828091    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tvkhd\""
Sep 17 23:12:42.828995 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.828979    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:12:42.829529 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.829513    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 23:12:42.834733 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.834716    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:12:42.834842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.834717    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:12:42.849820 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.849800    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"]
Sep 17 23:12:42.925616 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.925583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:42.938141 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.938112    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6g8m8"]
Sep 17 23:12:42.940492 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.940472    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:42.945168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.945150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:12:42.945393 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.945375    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z85sl\""
Sep 17 23:12:42.945693 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.945675    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:12:42.946527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.946508    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:12:42.995800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.995770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.995800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.995803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.996028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.995824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpxn\" (UniqueName: \"kubernetes.io/projected/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-api-access-zwpxn\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.996028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.995920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.996028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.995961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:42.996028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:42.996000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.045537 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.045445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:43.096354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-tls\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-accelerators-collector-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-sys\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.096527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpxn\" (UniqueName: \"kubernetes.io/projected/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-api-access-zwpxn\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-wtmp\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:43.096551    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwb5\" (UniqueName: \"kubernetes.io/projected/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-kube-api-access-gmwb5\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:12:43.096725    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls podName:fbc7d813-a53e-4f93-8b3e-e396a304a7bb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:43.596699558 +0000 UTC m=+118.404810699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-xtxxd" (UID: "fbc7d813-a53e-4f93-8b3e-e396a304a7bb") : secret "kube-state-metrics-tls" not found
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-root\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.096847 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-textfile\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.097274 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.097274 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.096921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-metrics-client-ca\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.097426 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.097395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.097612 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.097591    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.097854 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.097826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.099825 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.099801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.108109 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.108085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpxn\" (UniqueName: \"kubernetes.io/projected/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-api-access-zwpxn\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.198728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-tls\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.198793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-accelerators-collector-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.198855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-sys\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.198899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.198967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-wtmp\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.199034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwb5\" (UniqueName: \"kubernetes.io/projected/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-kube-api-access-gmwb5\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.199075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-root\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.199114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-textfile\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.199150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-metrics-client-ca\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.200396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.199916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-metrics-client-ca\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.203649 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.203619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-wtmp\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.203834 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.203810    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-tls\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.204010 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.203985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-sys\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.204157 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.204093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-accelerators-collector-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.204255 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.204217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-textfile\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.204468 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.204221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-root\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.206461 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.206301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.212948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.212928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwb5\" (UniqueName: \"kubernetes.io/projected/80bfd48b-bdb8-4bae-a3d2-2289ee9b6647-kube-api-access-gmwb5\") pod \"node-exporter-6g8m8\" (UID: \"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647\") " pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.250488 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.250454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6g8m8"
Sep 17 23:12:43.262193 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:43.262163    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bfd48b_bdb8_4bae_a3d2_2289ee9b6647.slice/crio-f202546f7ea5d0815bf3af92f7d54f554522adec1dd487400e021342daaa61af WatchSource:0}: Error finding container f202546f7ea5d0815bf3af92f7d54f554522adec1dd487400e021342daaa61af: Status 404 returned error can't find the container with id f202546f7ea5d0815bf3af92f7d54f554522adec1dd487400e021342daaa61af
Sep 17 23:12:43.273361 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.273334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:43.376548 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.376458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:43.604175 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.604125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.606415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.606395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbc7d813-a53e-4f93-8b3e-e396a304a7bb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-xtxxd\" (UID: \"fbc7d813-a53e-4f93-8b3e-e396a304a7bb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.734527 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.734493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"
Sep 17 23:12:43.820823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:43.820786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.056940 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:44.056910    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc7d813_a53e_4f93_8b3e_e396a304a7bb.slice/crio-496af57a56b1a5a2be8eca0938cc55e6d8ee8463861c0bfe1c4ee89ffc500066 WatchSource:0}: Error finding container 496af57a56b1a5a2be8eca0938cc55e6d8ee8463861c0bfe1c4ee89ffc500066: Status 404 returned error can't find the container with id 496af57a56b1a5a2be8eca0938cc55e6d8ee8463861c0bfe1c4ee89ffc500066
Sep 17 23:12:44.061354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.061332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-xtxxd"]
Sep 17 23:12:44.072503 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.072478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd" event={"ID":"fbc7d813-a53e-4f93-8b3e-e396a304a7bb","Type":"ContainerStarted","Data":"496af57a56b1a5a2be8eca0938cc55e6d8ee8463861c0bfe1c4ee89ffc500066"}
Sep 17 23:12:44.073814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.073785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6g8m8" event={"ID":"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647","Type":"ContainerStarted","Data":"f202546f7ea5d0815bf3af92f7d54f554522adec1dd487400e021342daaa61af"}
Sep 17 23:12:44.213650 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.213611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.512866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.512826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.524085 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.524063    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:44.527354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.527338    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.535702 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.535684    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 23:12:44.536025 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536013    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l8dnv\""
Sep 17 23:12:44.536313 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536242    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 23:12:44.536313 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536280    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 23:12:44.536313 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536279    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 23:12:44.536524 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536297    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 23:12:44.536624 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 23:12:44.536838 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.536822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 23:12:44.537246 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.537230    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 23:12:44.548503 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.548481    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 23:12:44.561375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.561356    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:44.615963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.615827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.615963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.615870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.615963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.615905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.615963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.615958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4t9p\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.616466 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.616305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.717535 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.717505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.717701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.717579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.717701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.717604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4t9p\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.717701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.717631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718015 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.717991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718413 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.718413 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.719016 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.718991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.719350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.719316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.720032 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.720006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.721417 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.721360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.721989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.721947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.722445 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.722418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.722537 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.722490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.722913 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.722854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.723288 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.723266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.723968 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.723923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.724124 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.724100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.724281 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.724262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.732522 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.732502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4t9p\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p\") pod \"alertmanager-main-0\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:44.755230 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.755182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.843029 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:44.842955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:12:45.079285 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:45.079250    2559 generic.go:356] "Generic (PLEG): container finished" podID="80bfd48b-bdb8-4bae-a3d2-2289ee9b6647" containerID="9443560b8ed660e8a986563f302c2ed896d57fdd87e28031f73ad253b84f9c5c" exitCode=0
Sep 17 23:12:45.079460 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:45.079315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6g8m8" event={"ID":"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647","Type":"ContainerDied","Data":"9443560b8ed660e8a986563f302c2ed896d57fdd87e28031f73ad253b84f9c5c"}
Sep 17 23:12:45.136892 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:45.136856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:12:45.150325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:45.150301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:12:45.254200 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:45.254166    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d85391a_d846_4c45_9613_c31bed7ae20d.slice/crio-64bc19b5c249261018c3eae6d2ff7810c6d4aaf27a31a8306e1fa10c9381ee98 WatchSource:0}: Error finding container 64bc19b5c249261018c3eae6d2ff7810c6d4aaf27a31a8306e1fa10c9381ee98: Status 404 returned error can't find the container with id 64bc19b5c249261018c3eae6d2ff7810c6d4aaf27a31a8306e1fa10c9381ee98
Sep 17 23:12:46.088647 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.088604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd" event={"ID":"fbc7d813-a53e-4f93-8b3e-e396a304a7bb","Type":"ContainerStarted","Data":"c30bcb7cfd106e96597bf9d8e2ba1879ded54b8599d0fa4c13adf558fa53b7f8"}
Sep 17 23:12:46.089112 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.088699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd" event={"ID":"fbc7d813-a53e-4f93-8b3e-e396a304a7bb","Type":"ContainerStarted","Data":"f198cf0247875f88d27dfce2de10a976f9ad1866ac15394e0ac9119aa08b9f55"}
Sep 17 23:12:46.089112 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.088726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd" event={"ID":"fbc7d813-a53e-4f93-8b3e-e396a304a7bb","Type":"ContainerStarted","Data":"9fec0b8fc49d0979765b7e7625da820bf55860273e309afe2cb29fc944df2c3e"}
Sep 17 23:12:46.090981 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.090956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6g8m8" event={"ID":"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647","Type":"ContainerStarted","Data":"9d5cef4366ca83b46551193c480f89bf273d10df90a2b1aa1671b7ed140a91e6"}
Sep 17 23:12:46.091101 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.091003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6g8m8" event={"ID":"80bfd48b-bdb8-4bae-a3d2-2289ee9b6647","Type":"ContainerStarted","Data":"8793ce6fb224c3b4dc9f1872941376ba0507c645c5f9a33a4a831628d957ea9e"}
Sep 17 23:12:46.092121 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.092100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"64bc19b5c249261018c3eae6d2ff7810c6d4aaf27a31a8306e1fa10c9381ee98"}
Sep 17 23:12:46.116967 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.116913    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-xtxxd" podStartSLOduration=2.875390955 podStartE2EDuration="4.116892366s" totalImagesPullingTime="1.241501411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:46.116564078 +0000 UTC m=+120.924675270" watchObservedRunningTime="2026-09-17 23:12:46.116892366 +0000 UTC m=+120.925003517"
Sep 17 23:12:46.140786 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.140733    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6g8m8" podStartSLOduration=2.813581665 podStartE2EDuration="4.14071236s" totalImagesPullingTime="1.25616489s" totalInitContainerRuntime="70.965805ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:46.139762914 +0000 UTC m=+120.947874064" watchObservedRunningTime="2026-09-17 23:12:46.14071236 +0000 UTC m=+120.948823510"
Sep 17 23:12:46.273236 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.273171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:46.493129 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.493099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:46.659457 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:46.659381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:47.097195 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.097161    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3" exitCode=0
Sep 17 23:12:47.097585 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.097245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"}
Sep 17 23:12:47.334214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.334178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:47.406077 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.405990    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-565cbf5f97-g9gjw"]
Sep 17 23:12:47.409146 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.409125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.412140 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412107    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f706cobmqnqvc\""
Sep 17 23:12:47.412259 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412148    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 23:12:47.412259 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7vdrc\""
Sep 17 23:12:47.412259 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412199    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 23:12:47.412551 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412536    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 23:12:47.412686 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.412648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:12:47.431325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.431300    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-565cbf5f97-g9gjw"]
Sep 17 23:12:47.449513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmjl\" (UniqueName: \"kubernetes.io/projected/647a4953-eca7-4fd6-85d6-ffa639f85641-kube-api-access-plmjl\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449604 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-client-certs\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-metrics-server-audit-profiles\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449729 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/647a4953-eca7-4fd6-85d6-ffa639f85641-audit-log\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449729 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-client-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449799 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-tls\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.449799 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.449759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.458889 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.458864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:47.550363 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/647a4953-eca7-4fd6-85d6-ffa639f85641-audit-log\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-client-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-tls\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550534 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmjl\" (UniqueName: \"kubernetes.io/projected/647a4953-eca7-4fd6-85d6-ffa639f85641-kube-api-access-plmjl\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550766 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-client-certs\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550766 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-metrics-server-audit-profiles\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.550865 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.550815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/647a4953-eca7-4fd6-85d6-ffa639f85641-audit-log\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.551244 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.551211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.551634 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.551611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/647a4953-eca7-4fd6-85d6-ffa639f85641-metrics-server-audit-profiles\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.553378 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.553353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-tls\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.553480 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.553385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-client-ca-bundle\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.553537 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.553519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/647a4953-eca7-4fd6-85d6-ffa639f85641-secret-metrics-server-client-certs\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.565095 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.565071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmjl\" (UniqueName: \"kubernetes.io/projected/647a4953-eca7-4fd6-85d6-ffa639f85641-kube-api-access-plmjl\") pod \"metrics-server-565cbf5f97-g9gjw\" (UID: \"647a4953-eca7-4fd6-85d6-ffa639f85641\") " pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:47.718120 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:47.718086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:12:48.010866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:48.006384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-565cbf5f97-g9gjw"]
Sep 17 23:12:48.302231 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:48.302160    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647a4953_eca7_4fd6_85d6_ffa639f85641.slice/crio-6220d6d365800bfe93087f6201b373ab1ad8d2e6a0a57354dd142f7d3ed24e7d WatchSource:0}: Error finding container 6220d6d365800bfe93087f6201b373ab1ad8d2e6a0a57354dd142f7d3ed24e7d: Status 404 returned error can't find the container with id 6220d6d365800bfe93087f6201b373ab1ad8d2e6a0a57354dd142f7d3ed24e7d
Sep 17 23:12:49.108644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.108603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"}
Sep 17 23:12:49.108842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.108718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"}
Sep 17 23:12:49.108842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.108765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"}
Sep 17 23:12:49.108842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.108809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"}
Sep 17 23:12:49.108974 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.108850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"}
Sep 17 23:12:49.109935 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.109911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw" event={"ID":"647a4953-eca7-4fd6-85d6-ffa639f85641","Type":"ContainerStarted","Data":"6220d6d365800bfe93087f6201b373ab1ad8d2e6a0a57354dd142f7d3ed24e7d"}
Sep 17 23:12:49.260199 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.260160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:49.396249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.396159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:49.631428 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.631390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:49.965213 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:49.965176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.073396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:50.073363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.117547 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:50.117460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw" event={"ID":"647a4953-eca7-4fd6-85d6-ffa639f85641","Type":"ContainerStarted","Data":"58d23b011c4b7e077f0a754ac6518a0b0c6ddb58c4014079bddd69a8e7c2ff53"}
Sep 17 23:12:50.120796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:50.120772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerStarted","Data":"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"}
Sep 17 23:12:50.147184 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:50.147130    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw" podStartSLOduration=1.9524630379999999 podStartE2EDuration="3.147111631s" totalImagesPullingTime="1.194648593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:50.146888643 +0000 UTC m=+124.954999794" watchObservedRunningTime="2026-09-17 23:12:50.147111631 +0000 UTC m=+124.955222781"
Sep 17 23:12:50.195581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:50.195519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.609328784 podStartE2EDuration="6.195496312s" totalImagesPullingTime="3.564869765s" totalInitContainerRuntime="21.297763ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:50.194381825 +0000 UTC m=+125.002492977" watchObservedRunningTime="2026-09-17 23:12:50.195496312 +0000 UTC m=+125.003607463"
Sep 17 23:12:51.751142 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:51.751100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:51.885303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:51.885270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.200961 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.200927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.305168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.305137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.837739 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.837708    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-cvxr4"]
Sep 17 23:12:54.842032 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.842016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:12:54.845106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.845085    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9mx4l\""
Sep 17 23:12:54.845393 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.845374    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:12:54.845475 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.845397    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:12:54.854030 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.854010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-cvxr4"]
Sep 17 23:12:54.861790 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.861766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.906898 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.906873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqp2\" (UniqueName: \"kubernetes.io/projected/fc5d8827-4b1e-41b8-8b80-e8e9b130a33e-kube-api-access-cfqp2\") pod \"downloads-6594f4c75-cvxr4\" (UID: \"fc5d8827-4b1e-41b8-8b80-e8e9b130a33e\") " pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:12:54.959091 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:54.959068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.007945 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.007925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqp2\" (UniqueName: \"kubernetes.io/projected/fc5d8827-4b1e-41b8-8b80-e8e9b130a33e-kube-api-access-cfqp2\") pod \"downloads-6594f4c75-cvxr4\" (UID: \"fc5d8827-4b1e-41b8-8b80-e8e9b130a33e\") " pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:12:55.027391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.027358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqp2\" (UniqueName: \"kubernetes.io/projected/fc5d8827-4b1e-41b8-8b80-e8e9b130a33e-kube-api-access-cfqp2\") pod \"downloads-6594f4c75-cvxr4\" (UID: \"fc5d8827-4b1e-41b8-8b80-e8e9b130a33e\") " pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:12:55.150124 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.150059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:12:55.174053 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.174024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.311472 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.311444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:12:55.313520 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.313501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21037b83-b73d-4375-9b84-b018549bbc05-metrics-certs\") pod \"network-metrics-daemon-zmndj\" (UID: \"21037b83-b73d-4375-9b84-b018549bbc05\") " pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:12:55.434166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.434140    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-cvxr4"]
Sep 17 23:12:55.435973 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:55.435947    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5d8827_4b1e_41b8_8b80_e8e9b130a33e.slice/crio-cc8556a0f4d31bd54d1e471b68b849f0d3ecdb524279f21d047ee0bc949cee4c WatchSource:0}: Error finding container cc8556a0f4d31bd54d1e471b68b849f0d3ecdb524279f21d047ee0bc949cee4c: Status 404 returned error can't find the container with id cc8556a0f4d31bd54d1e471b68b849f0d3ecdb524279f21d047ee0bc949cee4c
Sep 17 23:12:55.473063 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.473043    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6c57\""
Sep 17 23:12:55.478255 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.478240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmndj"
Sep 17 23:12:55.523416 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.523370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.624986 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.624956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.755485 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.755428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmndj"]
Sep 17 23:12:55.758163 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:12:55.758141    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21037b83_b73d_4375_9b84_b018549bbc05.slice/crio-e0bea14cd4f75b1bd2f1404eacd174793832ebdd73c779340abfd43a2a6e0a0a WatchSource:0}: Error finding container e0bea14cd4f75b1bd2f1404eacd174793832ebdd73c779340abfd43a2a6e0a0a: Status 404 returned error can't find the container with id e0bea14cd4f75b1bd2f1404eacd174793832ebdd73c779340abfd43a2a6e0a0a
Sep 17 23:12:55.789808 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:55.789782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.009144 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:56.009068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.145062 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:56.145034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-cvxr4" event={"ID":"fc5d8827-4b1e-41b8-8b80-e8e9b130a33e","Type":"ContainerStarted","Data":"cc8556a0f4d31bd54d1e471b68b849f0d3ecdb524279f21d047ee0bc949cee4c"}
Sep 17 23:12:56.146425 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:56.146393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmndj" event={"ID":"21037b83-b73d-4375-9b84-b018549bbc05","Type":"ContainerStarted","Data":"e0bea14cd4f75b1bd2f1404eacd174793832ebdd73c779340abfd43a2a6e0a0a"}
Sep 17 23:12:56.772174 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:56.772146    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.974378 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:56.974348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.152670 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.152555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmndj" event={"ID":"21037b83-b73d-4375-9b84-b018549bbc05","Type":"ContainerStarted","Data":"97eb389a906b85616a3a4738132353aa261dbd5a281033b2cc407a333a420d3b"}
Sep 17 23:12:57.152670 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.152618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmndj" event={"ID":"21037b83-b73d-4375-9b84-b018549bbc05","Type":"ContainerStarted","Data":"47d3f026eabfafeb3ac34bd40f2231d77e9f8eccf4a68d3f742621f070e5eae7"}
Sep 17 23:12:57.176137 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.176089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zmndj" podStartSLOduration=131.193108885 podStartE2EDuration="2m12.176070725s" totalImagesPullingTime="982.96184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:57.174996762 +0000 UTC m=+131.983107913" watchObservedRunningTime="2026-09-17 23:12:57.176070725 +0000 UTC m=+131.984181872"
Sep 17 23:12:57.180175 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.180142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.376573 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.376533    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.578006 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.577970    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.870117 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.870021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.971983 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:57.971951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.186246 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:58.186217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.306913 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:12:58.306880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.163726 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.163683    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerName="registry" containerID="cri-o://ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e" gracePeriod=30
Sep 17 23:13:00.202727 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.202692    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.334562 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.334524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.422900 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.422846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:13:00.556233 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556247    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pcbb\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556451    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556476    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556514    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets\") pod \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\" (UID: \"1fcc5758-af5e-4ff1-b9b5-a644331433b5\") "
Sep 17 23:13:00.556948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556880    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:00.557057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.556966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:00.559111 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.559069    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:00.559111 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.559088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:00.559644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.559606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:00.560000 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.559972    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:00.560160 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.560137    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb" (OuterVolumeSpecName: "kube-api-access-4pcbb") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "kube-api-access-4pcbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:00.568079 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.568047    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fcc5758-af5e-4ff1-b9b5-a644331433b5" (UID: "1fcc5758-af5e-4ff1-b9b5-a644331433b5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:13:00.657975 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.657939    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-trusted-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.657975 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.657971    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-bound-sa-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.657975 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.657980    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcbb\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-kube-api-access-4pcbb\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.658148 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.657992    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-certificates\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.658148 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.658000    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fcc5758-af5e-4ff1-b9b5-a644331433b5-ca-trust-extracted\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.658148 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.658009    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-installation-pull-secrets\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.658148 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.658018    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fcc5758-af5e-4ff1-b9b5-a644331433b5-image-registry-private-configuration\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:00.658148 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:00.658027    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fcc5758-af5e-4ff1-b9b5-a644331433b5-registry-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:01.172381 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.172334    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerID="ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e" exitCode=0
Sep 17 23:13:01.172381 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.172380    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5846d65468-dz2vm"
Sep 17 23:13:01.172881 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.172404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" event={"ID":"1fcc5758-af5e-4ff1-b9b5-a644331433b5","Type":"ContainerDied","Data":"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"}
Sep 17 23:13:01.172881 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.172464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5846d65468-dz2vm" event={"ID":"1fcc5758-af5e-4ff1-b9b5-a644331433b5","Type":"ContainerDied","Data":"cfed52b8a3aaedb305736397bfa40706142bbf8b325cb30c856ae0595e5f4b67"}
Sep 17 23:13:01.172881 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.172507    2559 scope.go:117] "RemoveContainer" containerID="ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"
Sep 17 23:13:01.182368 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.182346    2559 scope.go:117] "RemoveContainer" containerID="ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"
Sep 17 23:13:01.182682 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:13:01.182634    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e\": container with ID starting with ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e not found: ID does not exist" containerID="ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"
Sep 17 23:13:01.182760 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.182692    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e"} err="failed to get container status \"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e\": rpc error: code = NotFound desc = could not find container \"ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e\": container with ID starting with ddf097ffddb20c1bcaf8b4831614808221dc03e0a78f6c125280df239c1dc18e not found: ID does not exist"
Sep 17 23:13:01.197575 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.197551    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:13:01.202523 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.202499    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5846d65468-dz2vm"]
Sep 17 23:13:01.649794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:01.649764    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" path="/var/lib/kubelet/pods/1fcc5758-af5e-4ff1-b9b5-a644331433b5/volumes"
Sep 17 23:13:03.183350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:03.183321    2559 generic.go:356] "Generic (PLEG): container finished" podID="67517d31-1446-41a9-92ca-b6869abee076" containerID="ff30dd1b1f6f038463cab1a3f6d037e7a7c7cee3ce8d18cad1ea7894d0940d6e" exitCode=0
Sep 17 23:13:03.183751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:03.183356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8" event={"ID":"67517d31-1446-41a9-92ca-b6869abee076","Type":"ContainerDied","Data":"ff30dd1b1f6f038463cab1a3f6d037e7a7c7cee3ce8d18cad1ea7894d0940d6e"}
Sep 17 23:13:03.183751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:03.183703    2559 scope.go:117] "RemoveContainer" containerID="ff30dd1b1f6f038463cab1a3f6d037e7a7c7cee3ce8d18cad1ea7894d0940d6e"
Sep 17 23:13:04.196751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.196704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-drfk8" event={"ID":"67517d31-1446-41a9-92ca-b6869abee076","Type":"ContainerStarted","Data":"4bcfd3d8133b5f4788713ceec79eb76e12f38c3788324c36bd1e1b32940277f1"}
Sep 17 23:13:04.829398 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.828796    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:04.829398 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.829218    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerName="registry"
Sep 17 23:13:04.829398 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.829236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerName="registry"
Sep 17 23:13:04.829398 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.829301    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcc5758-af5e-4ff1-b9b5-a644331433b5" containerName="registry"
Sep 17 23:13:04.833939 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.833914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.837360 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.837339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:13:04.837468 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.837448    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:13:04.838504 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.838463    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:13:04.838633 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.838528    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2kpl6\""
Sep 17 23:13:04.838784 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.838765    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:13:04.839374 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.839352    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:13:04.845120 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.845099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:04.895163 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shld\" (UniqueName: \"kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.895252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.895252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.895336 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.895336 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.895336 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.895331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996385 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shld\" (UniqueName: \"kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996385 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.996645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.996485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.997374 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.997348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.997496 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.997378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.997496 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.997437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.999302 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.999271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:04.999415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:04.999385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:05.005370 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:05.005346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shld\" (UniqueName: \"kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld\") pod \"console-5fbb9f9446-gllgr\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") " pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:05.145408 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:05.145319    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:07.718325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:07.718289    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:13:07.718790 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:07.718360    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:13:09.158113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:09.158070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:09.262576 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:09.262533    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:10.880675 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:10.880632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:10.980340 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:10.980306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:11.070857 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:11.070824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:12.023113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.023089    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:13:12.026419 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.026403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.081495 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.081472    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:13:12.091814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.091786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:13:12.161076 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161076 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161076 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161265 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161265 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161265 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.161265 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.161180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.209404 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.209372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:12.212852 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:13:12.212824    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f253d5e_6dda_40ac_9241_63dad549bfef.slice/crio-ec08d3096776352122204222d8d309496790674a7bdbc1d0c8bed241c2a95f3f WatchSource:0}: Error finding container ec08d3096776352122204222d8d309496790674a7bdbc1d0c8bed241c2a95f3f: Status 404 returned error can't find the container with id ec08d3096776352122204222d8d309496790674a7bdbc1d0c8bed241c2a95f3f
Sep 17 23:13:12.233284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.233248    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-cvxr4" event={"ID":"fc5d8827-4b1e-41b8-8b80-e8e9b130a33e","Type":"ContainerStarted","Data":"5488d8bc11c3a2323c4213da84a68c6f1cb3f5a838924ec996e52b2ed88f3979"}
Sep 17 23:13:12.233728 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.233681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:13:12.235084 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.235061    2559 patch_prober.go:28] interesting pod/downloads-6594f4c75-cvxr4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused" start-of-body=
Sep 17 23:13:12.235187 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.235102    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-cvxr4" podUID="fc5d8827-4b1e-41b8-8b80-e8e9b130a33e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 23:13:12.235927 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.235902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbb9f9446-gllgr" event={"ID":"4f253d5e-6dda-40ac-9241-63dad549bfef","Type":"ContainerStarted","Data":"ec08d3096776352122204222d8d309496790674a7bdbc1d0c8bed241c2a95f3f"}
Sep 17 23:13:12.262207 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262334 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262334 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262448 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262554 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.262591 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.263051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.262975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.263476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.263365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.263476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.263453    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.263612 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.263560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.264929 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.264903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.265141 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.265119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.273036 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.272989    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-cvxr4" podStartSLOduration=1.6983096340000001 podStartE2EDuration="18.272970887s" totalImagesPullingTime="16.574661253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:12.271634595 +0000 UTC m=+147.079745747" watchObservedRunningTime="2026-09-17 23:13:12.272970887 +0000 UTC m=+147.081082040"
Sep 17 23:13:12.276866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.276844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm\") pod \"console-7f8fb5bc46-dkpcx\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") " pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.295688 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.295641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:12.379010 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.378960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:12.413170 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.413098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:12.710512 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:12.710471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:13:12.715916 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:13:12.715881    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a97e14_7bb0_4260_a129_5dfee87aeef9.slice/crio-63baaa821e452a6fff5e0a46d650e40b173023136ea82daecf852540b0b8b006 WatchSource:0}: Error finding container 63baaa821e452a6fff5e0a46d650e40b173023136ea82daecf852540b0b8b006: Status 404 returned error can't find the container with id 63baaa821e452a6fff5e0a46d650e40b173023136ea82daecf852540b0b8b006
Sep 17 23:13:13.251153 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:13.251113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fb5bc46-dkpcx" event={"ID":"36a97e14-7bb0-4260-a129-5dfee87aeef9","Type":"ContainerStarted","Data":"63baaa821e452a6fff5e0a46d650e40b173023136ea82daecf852540b0b8b006"}
Sep 17 23:13:13.263221 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:13.263189    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-cvxr4"
Sep 17 23:13:15.131214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:15.131174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:15.242909 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:15.242836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:16.597132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:16.597089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:16.685822 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:16.685745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:20.289491 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:20.289427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbb9f9446-gllgr" event={"ID":"4f253d5e-6dda-40ac-9241-63dad549bfef","Type":"ContainerStarted","Data":"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"}
Sep 17 23:13:20.291284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:20.291253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fb5bc46-dkpcx" event={"ID":"36a97e14-7bb0-4260-a129-5dfee87aeef9","Type":"ContainerStarted","Data":"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"}
Sep 17 23:13:20.312646 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:20.312585    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fbb9f9446-gllgr" podStartSLOduration=8.857399023 podStartE2EDuration="16.312564202s" totalImagesPullingTime="7.455165179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:20.312343441 +0000 UTC m=+155.120454591" watchObservedRunningTime="2026-09-17 23:13:20.312564202 +0000 UTC m=+155.120675354"
Sep 17 23:13:20.333831 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:20.333773    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f8fb5bc46-dkpcx" podStartSLOduration=2.396795979 podStartE2EDuration="9.333753995s" totalImagesPullingTime="6.936958016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:20.332578789 +0000 UTC m=+155.140689939" watchObservedRunningTime="2026-09-17 23:13:20.333753995 +0000 UTC m=+155.141865147"
Sep 17 23:13:21.440068 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:21.440034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:21.538296 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:21.538266    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:22.414034 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:22.413992    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:22.414245 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:22.414070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:22.419026 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:22.419000    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:23.307603 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:23.307570    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:13:23.363051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:23.363017    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:23.495422 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:23.495374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:23.595396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:23.595292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:23.668190 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:23.668145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:25.145973 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:25.145929    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:26.316796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:26.316758    2559 generic.go:356] "Generic (PLEG): container finished" podID="95e5e3d6-dbcb-4de2-a4de-a26a8c94c301" containerID="dea9ef7c87c29e09cc30761c0d98961a96ea2e61b91c4e334ca358c65e9cf3fe" exitCode=0
Sep 17 23:13:26.317226 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:26.316834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7" event={"ID":"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301","Type":"ContainerDied","Data":"dea9ef7c87c29e09cc30761c0d98961a96ea2e61b91c4e334ca358c65e9cf3fe"}
Sep 17 23:13:26.317226 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:26.317188    2559 scope.go:117] "RemoveContainer" containerID="dea9ef7c87c29e09cc30761c0d98961a96ea2e61b91c4e334ca358c65e9cf3fe"
Sep 17 23:13:27.322355 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:27.322321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-gpvx7" event={"ID":"95e5e3d6-dbcb-4de2-a4de-a26a8c94c301","Type":"ContainerStarted","Data":"8c21ffa0b87976432cc62dc808f5e8700c69a98b553b049349ec105317813963"}
Sep 17 23:13:27.724245 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:27.724216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:13:27.728129 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:27.728105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-565cbf5f97-g9gjw"
Sep 17 23:13:45.138435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:45.138352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:45.227718 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:45.227681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:48.383392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:48.383347    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fbb9f9446-gllgr" podUID="4f253d5e-6dda-40ac-9241-63dad549bfef" containerName="console" containerID="cri-o://9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1" gracePeriod=15
Sep 17 23:13:49.172262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.172225    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:49.289286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289202    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289268    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289298    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289327    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7shld\" (UniqueName: \"kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config\") pod \"4f253d5e-6dda-40ac-9241-63dad549bfef\" (UID: \"4f253d5e-6dda-40ac-9241-63dad549bfef\") "
Sep 17 23:13:49.289859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289830    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:49.289859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca" (OuterVolumeSpecName: "service-ca") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:49.289969 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.289849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config" (OuterVolumeSpecName: "console-config") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:49.291680 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.291640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:49.291777 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.291704    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld" (OuterVolumeSpecName: "kube-api-access-7shld") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "kube-api-access-7shld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:49.291777 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.291733    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4f253d5e-6dda-40ac-9241-63dad549bfef" (UID: "4f253d5e-6dda-40ac-9241-63dad549bfef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:49.390318 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390278    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.390318 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390311    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-service-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.390318 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390321    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-oauth-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.390318 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390330    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shld\" (UniqueName: \"kubernetes.io/projected/4f253d5e-6dda-40ac-9241-63dad549bfef-kube-api-access-7shld\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.390827 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390339    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f253d5e-6dda-40ac-9241-63dad549bfef-console-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.390827 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.390348    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f253d5e-6dda-40ac-9241-63dad549bfef-console-oauth-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:13:49.408944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.408915    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f253d5e-6dda-40ac-9241-63dad549bfef" containerID="9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1" exitCode=0
Sep 17 23:13:49.409106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.408963    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbb9f9446-gllgr"
Sep 17 23:13:49.409106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.408995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbb9f9446-gllgr" event={"ID":"4f253d5e-6dda-40ac-9241-63dad549bfef","Type":"ContainerDied","Data":"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"}
Sep 17 23:13:49.409106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.409048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbb9f9446-gllgr" event={"ID":"4f253d5e-6dda-40ac-9241-63dad549bfef","Type":"ContainerDied","Data":"ec08d3096776352122204222d8d309496790674a7bdbc1d0c8bed241c2a95f3f"}
Sep 17 23:13:49.409106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.409069    2559 scope.go:117] "RemoveContainer" containerID="9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"
Sep 17 23:13:49.418817 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.418793    2559 scope.go:117] "RemoveContainer" containerID="9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"
Sep 17 23:13:49.419132 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:13:49.419107    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1\": container with ID starting with 9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1 not found: ID does not exist" containerID="9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"
Sep 17 23:13:49.419193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.419145    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1"} err="failed to get container status \"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1\": rpc error: code = NotFound desc = could not find container \"9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1\": container with ID starting with 9abd74a1c5466a07d62c5a2403e4ce9e3d0b748e9b8e95b266f80afbdd25a5b1 not found: ID does not exist"
Sep 17 23:13:49.438596 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.438559    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:49.442669 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.442625    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fbb9f9446-gllgr"]
Sep 17 23:13:49.649323 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:13:49.649236    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f253d5e-6dda-40ac-9241-63dad549bfef" path="/var/lib/kubelet/pods/4f253d5e-6dda-40ac-9241-63dad549bfef/volumes"
Sep 17 23:14:02.592420 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:02.592381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:02.808934 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:02.808898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:03.648724 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.648693    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:03.649116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.648978    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="alertmanager" containerID="cri-o://ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898" gracePeriod=120
Sep 17 23:14:03.649116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.649047    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-metric" containerID="cri-o://7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894" gracePeriod=120
Sep 17 23:14:03.649116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.649069    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-web" containerID="cri-o://da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e" gracePeriod=120
Sep 17 23:14:03.649116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.649092    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="config-reloader" containerID="cri-o://9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d" gracePeriod=120
Sep 17 23:14:03.649319 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.649146    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy" containerID="cri-o://e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960" gracePeriod=120
Sep 17 23:14:03.649319 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:03.649129    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="prom-label-proxy" containerID="cri-o://e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194" gracePeriod=120
Sep 17 23:14:04.474119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474078    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194" exitCode=0
Sep 17 23:14:04.474119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474109    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960" exitCode=0
Sep 17 23:14:04.474119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474118    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d" exitCode=0
Sep 17 23:14:04.474119 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474125    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898" exitCode=0
Sep 17 23:14:04.474397 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"}
Sep 17 23:14:04.474397 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474254    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"}
Sep 17 23:14:04.474397 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"}
Sep 17 23:14:04.474397 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.474306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"}
Sep 17 23:14:04.914620 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:04.914591    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.031427 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031427 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031427 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031395    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4t9p\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031427 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031637    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031767    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.031814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031809    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.032159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031834    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.032159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031906    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets\") pod \"8d85391a-d846-4c45-9613-c31bed7ae20d\" (UID: \"8d85391a-d846-4c45-9613-c31bed7ae20d\") "
Sep 17 23:14:05.032159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.031985    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:14:05.032159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.032040    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:05.032363 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.032254    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:05.032363 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.032337    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.032472 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.032362    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d85391a-d846-4c45-9613-c31bed7ae20d-metrics-client-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.032472 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.032379    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-alertmanager-main-db\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.034452 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.034415    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.034948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.034916    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume" (OuterVolumeSpecName: "config-volume") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.035301 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.035243    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p" (OuterVolumeSpecName: "kube-api-access-h4t9p") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "kube-api-access-h4t9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:05.035443 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.035343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.035443 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.035428    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.035724 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.035685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.036005 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.035983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:05.036280 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.036258    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out" (OuterVolumeSpecName: "config-out") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:14:05.039064 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.039034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.046040 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.046014    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config" (OuterVolumeSpecName: "web-config") pod "8d85391a-d846-4c45-9613-c31bed7ae20d" (UID: "8d85391a-d846-4c45-9613-c31bed7ae20d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:05.133039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.132999    2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-cluster-tls-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133032    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d85391a-d846-4c45-9613-c31bed7ae20d-config-out\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133043    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-main-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133054    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133064    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-tls-assets\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133072    2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-config-volume\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133082    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-web-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133091    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4t9p\" (UniqueName: \"kubernetes.io/projected/8d85391a-d846-4c45-9613-c31bed7ae20d-kube-api-access-h4t9p\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133100    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.133286 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.133110    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d85391a-d846-4c45-9613-c31bed7ae20d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:05.481253 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481213    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894" exitCode=0
Sep 17 23:14:05.481253 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481249    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerID="da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e" exitCode=0
Sep 17 23:14:05.481517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"}
Sep 17 23:14:05.481517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481305    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.481517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"}
Sep 17 23:14:05.481517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d85391a-d846-4c45-9613-c31bed7ae20d","Type":"ContainerDied","Data":"64bc19b5c249261018c3eae6d2ff7810c6d4aaf27a31a8306e1fa10c9381ee98"}
Sep 17 23:14:05.481517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.481465    2559 scope.go:117] "RemoveContainer" containerID="e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"
Sep 17 23:14:05.489823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.489803    2559 scope.go:117] "RemoveContainer" containerID="7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"
Sep 17 23:14:05.497222 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.497201    2559 scope.go:117] "RemoveContainer" containerID="e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"
Sep 17 23:14:05.504768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.504741    2559 scope.go:117] "RemoveContainer" containerID="da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"
Sep 17 23:14:05.507959 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.507930    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:05.513506 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.513476    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:05.513826 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.513804    2559 scope.go:117] "RemoveContainer" containerID="9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"
Sep 17 23:14:05.521709 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.521679    2559 scope.go:117] "RemoveContainer" containerID="ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"
Sep 17 23:14:05.529450 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.529428    2559 scope.go:117] "RemoveContainer" containerID="0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"
Sep 17 23:14:05.537641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.537621    2559 scope.go:117] "RemoveContainer" containerID="e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"
Sep 17 23:14:05.538029 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.538005    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194\": container with ID starting with e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194 not found: ID does not exist" containerID="e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"
Sep 17 23:14:05.538105 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538040    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"} err="failed to get container status \"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194\": rpc error: code = NotFound desc = could not find container \"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194\": container with ID starting with e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194 not found: ID does not exist"
Sep 17 23:14:05.538105 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538062    2559 scope.go:117] "RemoveContainer" containerID="7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"
Sep 17 23:14:05.538346 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.538329    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894\": container with ID starting with 7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894 not found: ID does not exist" containerID="7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"
Sep 17 23:14:05.538425 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538351    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"} err="failed to get container status \"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894\": rpc error: code = NotFound desc = could not find container \"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894\": container with ID starting with 7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894 not found: ID does not exist"
Sep 17 23:14:05.538425 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538371    2559 scope.go:117] "RemoveContainer" containerID="e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"
Sep 17 23:14:05.538617 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.538601    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960\": container with ID starting with e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960 not found: ID does not exist" containerID="e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"
Sep 17 23:14:05.538651 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538622    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"} err="failed to get container status \"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960\": rpc error: code = NotFound desc = could not find container \"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960\": container with ID starting with e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960 not found: ID does not exist"
Sep 17 23:14:05.538651 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538637    2559 scope.go:117] "RemoveContainer" containerID="da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"
Sep 17 23:14:05.538883 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.538865    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e\": container with ID starting with da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e not found: ID does not exist" containerID="da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"
Sep 17 23:14:05.538924 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538889    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"} err="failed to get container status \"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e\": rpc error: code = NotFound desc = could not find container \"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e\": container with ID starting with da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e not found: ID does not exist"
Sep 17 23:14:05.538924 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.538904    2559 scope.go:117] "RemoveContainer" containerID="9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"
Sep 17 23:14:05.539121 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.539106    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d\": container with ID starting with 9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d not found: ID does not exist" containerID="9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"
Sep 17 23:14:05.539162 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539124    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"} err="failed to get container status \"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d\": rpc error: code = NotFound desc = could not find container \"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d\": container with ID starting with 9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d not found: ID does not exist"
Sep 17 23:14:05.539162 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539136    2559 scope.go:117] "RemoveContainer" containerID="ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"
Sep 17 23:14:05.539342 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.539327    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898\": container with ID starting with ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898 not found: ID does not exist" containerID="ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"
Sep 17 23:14:05.539400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539345    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"} err="failed to get container status \"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898\": rpc error: code = NotFound desc = could not find container \"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898\": container with ID starting with ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898 not found: ID does not exist"
Sep 17 23:14:05.539400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539358    2559 scope.go:117] "RemoveContainer" containerID="0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"
Sep 17 23:14:05.539581 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:05.539565    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3\": container with ID starting with 0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3 not found: ID does not exist" containerID="0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"
Sep 17 23:14:05.539639 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539584    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"} err="failed to get container status \"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3\": rpc error: code = NotFound desc = could not find container \"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3\": container with ID starting with 0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3 not found: ID does not exist"
Sep 17 23:14:05.539639 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539596    2559 scope.go:117] "RemoveContainer" containerID="e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"
Sep 17 23:14:05.539816 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539799    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194"} err="failed to get container status \"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194\": rpc error: code = NotFound desc = could not find container \"e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194\": container with ID starting with e7c1ba60e3b62a4e42656b0edfa608fddf14685289b3d68712a2a426f7dc8194 not found: ID does not exist"
Sep 17 23:14:05.539854 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.539815    2559 scope.go:117] "RemoveContainer" containerID="7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"
Sep 17 23:14:05.540027 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540012    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894"} err="failed to get container status \"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894\": rpc error: code = NotFound desc = could not find container \"7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894\": container with ID starting with 7a8aeac45303dbf18cabbc3352e49b643acc5cc80083407722887670b2180894 not found: ID does not exist"
Sep 17 23:14:05.540069 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540028    2559 scope.go:117] "RemoveContainer" containerID="e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"
Sep 17 23:14:05.540215 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540200    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960"} err="failed to get container status \"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960\": rpc error: code = NotFound desc = could not find container \"e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960\": container with ID starting with e369b95348077752aea6b5a5c2e6741a222b7fe414767f8438385d1eae2c8960 not found: ID does not exist"
Sep 17 23:14:05.540254 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540214    2559 scope.go:117] "RemoveContainer" containerID="da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"
Sep 17 23:14:05.540423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540407    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e"} err="failed to get container status \"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e\": rpc error: code = NotFound desc = could not find container \"da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e\": container with ID starting with da26ee0af2d0ee045a56c67400abf67e41d785f34ba931cd01fcdc1a3625a65e not found: ID does not exist"
Sep 17 23:14:05.540465 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540425    2559 scope.go:117] "RemoveContainer" containerID="9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"
Sep 17 23:14:05.540612 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540598    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d"} err="failed to get container status \"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d\": rpc error: code = NotFound desc = could not find container \"9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d\": container with ID starting with 9bed0983ac76cd296c83356c620a0ee8d2c810c9fac6265a5d37c1172cb18b1d not found: ID does not exist"
Sep 17 23:14:05.540673 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540612    2559 scope.go:117] "RemoveContainer" containerID="ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"
Sep 17 23:14:05.540826 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540809    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898"} err="failed to get container status \"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898\": rpc error: code = NotFound desc = could not find container \"ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898\": container with ID starting with ae57b840084a5bad672919d86850559dfda09d420b8fc156b7609e5c5a43d898 not found: ID does not exist"
Sep 17 23:14:05.540871 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540826    2559 scope.go:117] "RemoveContainer" containerID="0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"
Sep 17 23:14:05.541010 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.540994    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3"} err="failed to get container status \"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3\": rpc error: code = NotFound desc = could not find container \"0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3\": container with ID starting with 0e875aa2147ea69d2f3f3c1af4d2047f94f813e90065881a5db5a5b20f9d5ef3 not found: ID does not exist"
Sep 17 23:14:05.544880 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.544857    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:05.545142 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545130    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="config-reloader"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545143    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="config-reloader"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545151    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f253d5e-6dda-40ac-9241-63dad549bfef" containerName="console"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f253d5e-6dda-40ac-9241-63dad549bfef" containerName="console"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545166    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545171    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545177    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="prom-label-proxy"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545182    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="prom-label-proxy"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545191    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-metric"
Sep 17 23:14:05.545193 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545196    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-metric"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545205    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="init-config-reloader"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545211    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="init-config-reloader"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545217    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-web"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545222    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-web"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545231    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="alertmanager"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="alertmanager"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="prom-label-proxy"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545306    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545313    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-metric"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545319    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="alertmanager"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545325    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f253d5e-6dda-40ac-9241-63dad549bfef" containerName="console"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545332    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="config-reloader"
Sep 17 23:14:05.545478 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.545338    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" containerName="kube-rbac-proxy-web"
Sep 17 23:14:05.549228 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.549206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.554123 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554102    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 23:14:05.554275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l8dnv\""
Sep 17 23:14:05.554371 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 23:14:05.554600 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554580    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 23:14:05.554706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 23:14:05.554706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554693    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 23:14:05.554929 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554910    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 23:14:05.555003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.554933    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 23:14:05.560457 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.560438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 23:14:05.565578 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.565554    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 23:14:05.568610 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.568585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:05.637500 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-web-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-config-out\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637719 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-config-volume\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.637930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.637920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzmn\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-kube-api-access-snzmn\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.649796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.649755    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d85391a-d846-4c45-9613-c31bed7ae20d" path="/var/lib/kubelet/pods/8d85391a-d846-4c45-9613-c31bed7ae20d/volumes"
Sep 17 23:14:05.739337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-config-volume\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzmn\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-kube-api-access-snzmn\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-web-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739993 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-config-out\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739993 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.739993 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.739732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.740272 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.740240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.740421 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.740380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e014c759-1848-4f73-938f-c789d4925569-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-config-volume\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742637 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742731 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.742865 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.742836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-web-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.743299 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.743277    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.743368 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.743276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.744096 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.744078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e014c759-1848-4f73-938f-c789d4925569-config-out\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.744892 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.744873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e014c759-1848-4f73-938f-c789d4925569-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.748440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.748421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzmn\" (UniqueName: \"kubernetes.io/projected/e014c759-1848-4f73-938f-c789d4925569-kube-api-access-snzmn\") pod \"alertmanager-main-0\" (UID: \"e014c759-1848-4f73-938f-c789d4925569\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:05.859402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:05.859357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:14:06.143553 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:06.143465    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:14:06.146892 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:14:06.146862    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode014c759_1848_4f73_938f_c789d4925569.slice/crio-2680655a639246ed7c2056c8f47dbedd4f791bfac9ed339af18d6f51f07db323 WatchSource:0}: Error finding container 2680655a639246ed7c2056c8f47dbedd4f791bfac9ed339af18d6f51f07db323: Status 404 returned error can't find the container with id 2680655a639246ed7c2056c8f47dbedd4f791bfac9ed339af18d6f51f07db323
Sep 17 23:14:06.487979 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:06.487939    2559 generic.go:356] "Generic (PLEG): container finished" podID="e014c759-1848-4f73-938f-c789d4925569" containerID="2969a0c9ccf47393d69faf3a3fdfdb1c05a8015a2910c0f34a201904534027df" exitCode=0
Sep 17 23:14:06.487979 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:06.487984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerDied","Data":"2969a0c9ccf47393d69faf3a3fdfdb1c05a8015a2910c0f34a201904534027df"}
Sep 17 23:14:06.488206 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:06.488041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"2680655a639246ed7c2056c8f47dbedd4f791bfac9ed339af18d6f51f07db323"}
Sep 17 23:14:07.494956 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.494920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"dba21fcb0016a221ca1910a9b2be96ebafa487567a723e58d873cb68291cfb24"}
Sep 17 23:14:07.495377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.494985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"ae9c6e86dff0a95ca04610df9375e5299b29af8774396a9d27c4aaf6ede973c9"}
Sep 17 23:14:07.495377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.495026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"a129fb56e605ff27d26fd6148a7b7e8b4b7859f2446d22c8da474c85a7a0eb8f"}
Sep 17 23:14:07.495377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.495057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"2b26768dbb019d7846bacefe318a67c3ee940468b7234d83d986f13c7cf6306b"}
Sep 17 23:14:07.495377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.495082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"54989f83fd9b55311ab028b8f436c47eb726ffd00890051b8961e743e5ee2e3b"}
Sep 17 23:14:07.495377 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.495116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e014c759-1848-4f73-938f-c789d4925569","Type":"ContainerStarted","Data":"c9082885af5d27552da00b5bbdc751d63dc1c6a0492b244e4e73e59b888068fc"}
Sep 17 23:14:07.530542 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.530486    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.515963135 podStartE2EDuration="2.530467808s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.504673ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:07.52871784 +0000 UTC m=+202.336828991" watchObservedRunningTime="2026-09-17 23:14:07.530467808 +0000 UTC m=+202.338578959"
Sep 17 23:14:07.710234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.710196    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"]
Sep 17 23:14:07.712615 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.712597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.715238 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.715209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-24s9l\""
Sep 17 23:14:07.715819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.715801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 23:14:07.715956 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.715842    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 23:14:07.716074 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.716059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 23:14:07.716248 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.716227    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 23:14:07.716654 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.716627    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 23:14:07.721672 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.721644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 23:14:07.733103 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.733080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"]
Sep 17 23:14:07.854635 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-metrics-client-ca\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854635 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854593    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-federate-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-serving-certs-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkhz\" (UniqueName: \"kubernetes.io/projected/0f4bcd70-35a7-4389-8ef1-18cac8a64259-kube-api-access-9dkhz\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.854864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.854851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955548 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-federate-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-serving-certs-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkhz\" (UniqueName: \"kubernetes.io/projected/0f4bcd70-35a7-4389-8ef1-18cac8a64259-kube-api-access-9dkhz\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.955821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.955779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-metrics-client-ca\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.956536 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.956507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-serving-certs-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.956656 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.956556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-metrics-client-ca\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.956779 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.956758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.958209 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.958179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-federate-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.958386 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.958353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.958674 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.958639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-telemeter-client-tls\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.958733 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.958693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f4bcd70-35a7-4389-8ef1-18cac8a64259-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:07.967339 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:07.967317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkhz\" (UniqueName: \"kubernetes.io/projected/0f4bcd70-35a7-4389-8ef1-18cac8a64259-kube-api-access-9dkhz\") pod \"telemeter-client-655f9b4bc6-tmsmn\" (UID: \"0f4bcd70-35a7-4389-8ef1-18cac8a64259\") " pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:08.023252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:08.023227    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"
Sep 17 23:14:08.302734 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:08.302712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn"]
Sep 17 23:14:08.303919 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:14:08.303887    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4bcd70_35a7_4389_8ef1_18cac8a64259.slice/crio-5efd1defb6bd9b686d65ac636772347ff90843947e3fdf44fb441ce74ace033f WatchSource:0}: Error finding container 5efd1defb6bd9b686d65ac636772347ff90843947e3fdf44fb441ce74ace033f: Status 404 returned error can't find the container with id 5efd1defb6bd9b686d65ac636772347ff90843947e3fdf44fb441ce74ace033f
Sep 17 23:14:08.500701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:08.500669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn" event={"ID":"0f4bcd70-35a7-4389-8ef1-18cac8a64259","Type":"ContainerStarted","Data":"5efd1defb6bd9b686d65ac636772347ff90843947e3fdf44fb441ce74ace033f"}
Sep 17 23:14:10.511848 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.511808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn" event={"ID":"0f4bcd70-35a7-4389-8ef1-18cac8a64259","Type":"ContainerStarted","Data":"9677ff051cf6cfffb6809befea10756f4685a822e26a5fa11f343bac266bb3a4"}
Sep 17 23:14:10.512212 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.511888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn" event={"ID":"0f4bcd70-35a7-4389-8ef1-18cac8a64259","Type":"ContainerStarted","Data":"190e3c0ead353dd7dfb81f4c384204fc37efebeed4f10115ecd33b5d5392d7c3"}
Sep 17 23:14:10.512212 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.511914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn" event={"ID":"0f4bcd70-35a7-4389-8ef1-18cac8a64259","Type":"ContainerStarted","Data":"8874239d97111577831a819870430a4c6f4cc8caecc7d0c80d0e27c5884b51ba"}
Sep 17 23:14:10.539059 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.539007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-655f9b4bc6-tmsmn" podStartSLOduration=1.976191328 podStartE2EDuration="3.538975027s" totalImagesPullingTime="1.562783699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:10.536249944 +0000 UTC m=+205.344361127" watchObservedRunningTime="2026-09-17 23:14:10.538975027 +0000 UTC m=+205.347086178"
Sep 17 23:14:10.640932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.640906    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:14:10.643470 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.643447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.659272 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.659247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:14:10.733113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.733078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:10.775249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775249 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqmw\" (UniqueName: \"kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775340    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775459 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775459 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.775529 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.775486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.821496 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.821471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:10.876394 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876394 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqmw\" (UniqueName: \"kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.876563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.876531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.877214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.877169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.877214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.877178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.877318 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.877289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.877356 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.877340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.878967 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.878940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.879042 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.879026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.885800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.885778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqmw\" (UniqueName: \"kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw\") pod \"console-55d489b4cc-pbwbv\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") " pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:10.910342 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.910318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:10.953244 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:10.953216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:11.224089 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.224041    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:14:11.226534 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:14:11.226504    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedb801e_e40e_49b7_9424_2b85f7f25d83.slice/crio-74514b0bb528f9e79043c82498ea2b8160ec044cb125cd38ebd285bb50c0facc WatchSource:0}: Error finding container 74514b0bb528f9e79043c82498ea2b8160ec044cb125cd38ebd285bb50c0facc: Status 404 returned error can't find the container with id 74514b0bb528f9e79043c82498ea2b8160ec044cb125cd38ebd285bb50c0facc
Sep 17 23:14:11.447032 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.446998    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:11.517109 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.517042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d489b4cc-pbwbv" event={"ID":"dedb801e-e40e-49b7-9424-2b85f7f25d83","Type":"ContainerStarted","Data":"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"}
Sep 17 23:14:11.517109 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.517094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d489b4cc-pbwbv" event={"ID":"dedb801e-e40e-49b7-9424-2b85f7f25d83","Type":"ContainerStarted","Data":"74514b0bb528f9e79043c82498ea2b8160ec044cb125cd38ebd285bb50c0facc"}
Sep 17 23:14:11.519307 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.519277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:11.540364 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:11.540322    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55d489b4cc-pbwbv" podStartSLOduration=1.540306611 podStartE2EDuration="1.540306611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:11.537805081 +0000 UTC m=+206.345916231" watchObservedRunningTime="2026-09-17 23:14:11.540306611 +0000 UTC m=+206.348417827"
Sep 17 23:14:20.954205 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:20.954156    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:20.954802 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:20.954235    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:20.959116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:20.959088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:21.563790 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:21.563756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:14:21.637576 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:21.637547    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:14:21.839000 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:21.838918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:21.953871 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:21.953843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:46.657964 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.657922    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f8fb5bc46-dkpcx" podUID="36a97e14-7bb0-4260-a129-5dfee87aeef9" containerName="console" containerID="cri-o://48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de" gracePeriod=15
Sep 17 23:14:46.907628 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.907602    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:14:46.979092 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979054    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979092 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979119    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979195    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979369 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config\") pod \"36a97e14-7bb0-4260-a129-5dfee87aeef9\" (UID: \"36a97e14-7bb0-4260-a129-5dfee87aeef9\") "
Sep 17 23:14:46.979631 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:46.979746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979712    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:46.979746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca" (OuterVolumeSpecName: "service-ca") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:46.979929 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:46.979988 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.979944    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config" (OuterVolumeSpecName: "console-config") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:46.981375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.981353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:46.981375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.981358    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm" (OuterVolumeSpecName: "kube-api-access-8d4wm") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "kube-api-access-8d4wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:46.981375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:46.981367    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "36a97e14-7bb0-4260-a129-5dfee87aeef9" (UID: "36a97e14-7bb0-4260-a129-5dfee87aeef9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:47.080269 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080231    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-oauth-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.080269 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080264    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.080269 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080276    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/36a97e14-7bb0-4260-a129-5dfee87aeef9-kube-api-access-8d4wm\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.080507 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080286    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-oauth-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.080507 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080295    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36a97e14-7bb0-4260-a129-5dfee87aeef9-service-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.080507 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.080305    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36a97e14-7bb0-4260-a129-5dfee87aeef9-console-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:14:47.660731 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.660702    2559 generic.go:356] "Generic (PLEG): container finished" podID="36a97e14-7bb0-4260-a129-5dfee87aeef9" containerID="48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de" exitCode=0
Sep 17 23:14:47.661166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.660745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fb5bc46-dkpcx" event={"ID":"36a97e14-7bb0-4260-a129-5dfee87aeef9","Type":"ContainerDied","Data":"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"}
Sep 17 23:14:47.661166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.660754    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8fb5bc46-dkpcx"
Sep 17 23:14:47.661166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.660782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8fb5bc46-dkpcx" event={"ID":"36a97e14-7bb0-4260-a129-5dfee87aeef9","Type":"ContainerDied","Data":"63baaa821e452a6fff5e0a46d650e40b173023136ea82daecf852540b0b8b006"}
Sep 17 23:14:47.661166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.660800    2559 scope.go:117] "RemoveContainer" containerID="48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"
Sep 17 23:14:47.668953 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.668942    2559 scope.go:117] "RemoveContainer" containerID="48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"
Sep 17 23:14:47.669226 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:14:47.669203    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de\": container with ID starting with 48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de not found: ID does not exist" containerID="48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"
Sep 17 23:14:47.669297 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.669239    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de"} err="failed to get container status \"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de\": rpc error: code = NotFound desc = could not find container \"48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de\": container with ID starting with 48fb5fdabcdf72619f4016fac3f6c5f0a1dfe834216cd929d29b8d5d893b66de not found: ID does not exist"
Sep 17 23:14:47.679742 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.679719    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:14:47.682924 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:47.682904    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f8fb5bc46-dkpcx"]
Sep 17 23:14:49.648800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:14:49.648769    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a97e14-7bb0-4260-a129-5dfee87aeef9" path="/var/lib/kubelet/pods/36a97e14-7bb0-4260-a129-5dfee87aeef9/volumes"
Sep 17 23:15:06.930415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:06.930363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:17.884707 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:17.884651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:17.994379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:17.994352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:21.668797 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.668766    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:15:21.669195 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.669177    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a97e14-7bb0-4260-a129-5dfee87aeef9" containerName="console"
Sep 17 23:15:21.669242 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.669198    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a97e14-7bb0-4260-a129-5dfee87aeef9" containerName="console"
Sep 17 23:15:21.669287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.669277    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a97e14-7bb0-4260-a129-5dfee87aeef9" containerName="console"
Sep 17 23:15:21.672474 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.672455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.685925 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.685902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:15:21.763340 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.763310    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:21.832396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7r2\" (UniqueName: \"kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.832682 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.832626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.902954 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.902928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:21.933592 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933682 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933682 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933759 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7r2\" (UniqueName: \"kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933759 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933759 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.933976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.933893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.934531 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.934500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.934655 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.934604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.934655 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.934604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.934806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.934757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.936030 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.936008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.936128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.936071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.944547 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.944526    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7r2\" (UniqueName: \"kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2\") pod \"console-75d94d4b7b-p6x6p\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") " pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:21.981045 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:21.981022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:22.259709 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:22.259679    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:15:22.797171 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:22.797143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d94d4b7b-p6x6p" event={"ID":"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1","Type":"ContainerStarted","Data":"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"}
Sep 17 23:15:22.797510 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:22.797186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d94d4b7b-p6x6p" event={"ID":"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1","Type":"ContainerStarted","Data":"d0171d7f5da63742ffca9655778f0f92844fcab0626c1daf162fc8af95ec446c"}
Sep 17 23:15:25.927228 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:25.927184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:29.885878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:29.885837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:29.959896 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:29.959865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:31.982037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:31.982005    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:31.982411 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:31.982060    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:31.986586 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:31.986562    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:32.008174 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:32.008122    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75d94d4b7b-p6x6p" podStartSLOduration=11.008105828 podStartE2EDuration="11.008105828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:22.824482686 +0000 UTC m=+277.632593837" watchObservedRunningTime="2026-09-17 23:15:32.008105828 +0000 UTC m=+286.816216980"
Sep 17 23:15:32.839902 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:32.839875    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:15:32.916283 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:32.916254    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:15:33.087256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:33.087221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:33.198638 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:33.198615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:40.215584 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:40.215542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:45.182769 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:45.182731    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:45.305395 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:45.305354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:45.551047 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:45.550976    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:15:57.935348 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:57.935290    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55d489b4cc-pbwbv" podUID="dedb801e-e40e-49b7-9424-2b85f7f25d83" containerName="console" containerID="cri-o://9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94" gracePeriod=15
Sep 17 23:15:58.167718 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.167696    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:15:58.207601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207593    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jqmw\" (UniqueName: \"kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207631    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.207859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle\") pod \"dedb801e-e40e-49b7-9424-2b85f7f25d83\" (UID: \"dedb801e-e40e-49b7-9424-2b85f7f25d83\") "
Sep 17 23:15:58.208066 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.207983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config" (OuterVolumeSpecName: "console-config") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:58.208303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.208260    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:58.208303 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.208289    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca" (OuterVolumeSpecName: "service-ca") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:58.208477 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.208348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:15:58.209856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.209832    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:15:58.209856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.209841    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw" (OuterVolumeSpecName: "kube-api-access-2jqmw") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "kube-api-access-2jqmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:15:58.210131 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.210106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dedb801e-e40e-49b7-9424-2b85f7f25d83" (UID: "dedb801e-e40e-49b7-9424-2b85f7f25d83"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:15:58.308321 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308301    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-oauth-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308321 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308320    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308328    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308337    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dedb801e-e40e-49b7-9424-2b85f7f25d83-service-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308345    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jqmw\" (UniqueName: \"kubernetes.io/projected/dedb801e-e40e-49b7-9424-2b85f7f25d83-kube-api-access-2jqmw\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308353    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.308440 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.308361    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dedb801e-e40e-49b7-9424-2b85f7f25d83-console-oauth-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:15:58.938375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.938337    2559 generic.go:356] "Generic (PLEG): container finished" podID="dedb801e-e40e-49b7-9424-2b85f7f25d83" containerID="9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94" exitCode=0
Sep 17 23:15:58.938768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.938387    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55d489b4cc-pbwbv"
Sep 17 23:15:58.938768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.938420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d489b4cc-pbwbv" event={"ID":"dedb801e-e40e-49b7-9424-2b85f7f25d83","Type":"ContainerDied","Data":"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"}
Sep 17 23:15:58.938768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.938475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55d489b4cc-pbwbv" event={"ID":"dedb801e-e40e-49b7-9424-2b85f7f25d83","Type":"ContainerDied","Data":"74514b0bb528f9e79043c82498ea2b8160ec044cb125cd38ebd285bb50c0facc"}
Sep 17 23:15:58.938768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.938495    2559 scope.go:117] "RemoveContainer" containerID="9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"
Sep 17 23:15:58.946460 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.946439    2559 scope.go:117] "RemoveContainer" containerID="9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"
Sep 17 23:15:58.946748 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:15:58.946728    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94\": container with ID starting with 9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94 not found: ID does not exist" containerID="9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"
Sep 17 23:15:58.946808 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.946756    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94"} err="failed to get container status \"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94\": rpc error: code = NotFound desc = could not find container \"9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94\": container with ID starting with 9f99d3af210968b3106db7caaaf7f190a151d252fc630557b49db48887bdba94 not found: ID does not exist"
Sep 17 23:15:58.964134 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.964112    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:15:58.968176 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:58.968154    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55d489b4cc-pbwbv"]
Sep 17 23:15:59.648488 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:15:59.648452    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedb801e-e40e-49b7-9424-2b85f7f25d83" path="/var/lib/kubelet/pods/dedb801e-e40e-49b7-9424-2b85f7f25d83/volumes"
Sep 17 23:16:38.611320 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:16:38.611229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:39.887570 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:16:39.887512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:02.238113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:02.238078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:02.331150 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:02.331099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:08.270239 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:08.270173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:08.764011 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:08.763977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:09.055932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:09.055844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:09.336265 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:09.336181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:09.587199 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:09.587100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:09.762357 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:09.762327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:09.979834 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:09.979802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:10.129109 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:10.129080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:10.292948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:10.292863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:10.515808 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:10.515777    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:10.702850 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:10.702817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:20.561313 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:20.561272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:59.708515 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:59.708470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:59.768929 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:17:59.768889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.886324 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.886277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.959588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.959560    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"]
Sep 17 23:18:25.959971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.959947    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedb801e-e40e-49b7-9424-2b85f7f25d83" containerName="console"
Sep 17 23:18:25.959971 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.959969    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedb801e-e40e-49b7-9424-2b85f7f25d83" containerName="console"
Sep 17 23:18:25.960113 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.960042    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedb801e-e40e-49b7-9424-2b85f7f25d83" containerName="console"
Sep 17 23:18:25.962875 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.962857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:25.965420 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.965397    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-2rnl9\""
Sep 17 23:18:25.965561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.965495    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:18:25.966106 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.966087    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 23:18:25.966204 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.966154    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:18:25.967433 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.967404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.974958 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.974936    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"]
Sep 17 23:18:25.987702 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.987679    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-f2qc4"]
Sep 17 23:18:25.990838 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.990818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:25.993621 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.993600    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 23:18:25.993866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:25.993845    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wvwbg\""
Sep 17 23:18:26.001877 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.001856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f2qc4"]
Sep 17 23:18:26.046299 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.046270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdf7f\" (UniqueName: \"kubernetes.io/projected/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-kube-api-access-cdf7f\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.046435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.046370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jq97\" (UniqueName: \"kubernetes.io/projected/bea51e17-c521-497d-b499-84b634e06cff-kube-api-access-8jq97\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.046499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.046432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.046499 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.046456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-data\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.061585 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.061557    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.147789 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.147725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.147789 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.147756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-data\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.147789 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.147783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdf7f\" (UniqueName: \"kubernetes.io/projected/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-kube-api-access-cdf7f\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.147982 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.147851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jq97\" (UniqueName: \"kubernetes.io/projected/bea51e17-c521-497d-b499-84b634e06cff-kube-api-access-8jq97\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.148051 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:18:26.148032    2559 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 23:18:26.148122 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:18:26.148109    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert podName:bea51e17-c521-497d-b499-84b634e06cff nodeName:}" failed. No retries permitted until 2026-09-17 23:18:26.648087641 +0000 UTC m=+461.456198783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert") pod "llmisvc-controller-manager-645f9c8fd6-jvsvq" (UID: "bea51e17-c521-497d-b499-84b634e06cff") : secret "llmisvc-webhook-server-cert" not found
Sep 17 23:18:26.148194 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.148174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-data\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.156748 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.156722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdf7f\" (UniqueName: \"kubernetes.io/projected/bc44ba4c-4048-4f38-b017-1f1a5c2ddda3-kube-api-access-cdf7f\") pod \"seaweedfs-64568bcd49-f2qc4\" (UID: \"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3\") " pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.157122 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.157100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jq97\" (UniqueName: \"kubernetes.io/projected/bea51e17-c521-497d-b499-84b634e06cff-kube-api-access-8jq97\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.301290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.301262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:26.428555 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.428466    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.577824 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.577795    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f2qc4"]
Sep 17 23:18:26.581234 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:18:26.581207    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc44ba4c_4048_4f38_b017_1f1a5c2ddda3.slice/crio-9e466df7b12516fc0109cc57e95b763d3cb5ffd4ab9c7e7b407bd033c0ce8943 WatchSource:0}: Error finding container 9e466df7b12516fc0109cc57e95b763d3cb5ffd4ab9c7e7b407bd033c0ce8943: Status 404 returned error can't find the container with id 9e466df7b12516fc0109cc57e95b763d3cb5ffd4ab9c7e7b407bd033c0ce8943
Sep 17 23:18:26.582484 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.582464    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:18:26.641591 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.641564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.651973 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.651950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.654151 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.654129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea51e17-c521-497d-b499-84b634e06cff-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-jvsvq\" (UID: \"bea51e17-c521-497d-b499-84b634e06cff\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.855409 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.855378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.875325 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.875294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:26.952516 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:26.952483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:27.032084 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:27.032052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:27.148536 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:27.148319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"]
Sep 17 23:18:27.150456 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:18:27.150431    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbea51e17_c521_497d_b499_84b634e06cff.slice/crio-374c14ca126ece4f97d8016da91ec99196fb8a3ab6be88cfed2c80b051af3eb8 WatchSource:0}: Error finding container 374c14ca126ece4f97d8016da91ec99196fb8a3ab6be88cfed2c80b051af3eb8: Status 404 returned error can't find the container with id 374c14ca126ece4f97d8016da91ec99196fb8a3ab6be88cfed2c80b051af3eb8
Sep 17 23:18:27.490906 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:27.490869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f2qc4" event={"ID":"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3","Type":"ContainerStarted","Data":"9e466df7b12516fc0109cc57e95b763d3cb5ffd4ab9c7e7b407bd033c0ce8943"}
Sep 17 23:18:27.492135 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:27.492112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq" event={"ID":"bea51e17-c521-497d-b499-84b634e06cff","Type":"ContainerStarted","Data":"374c14ca126ece4f97d8016da91ec99196fb8a3ab6be88cfed2c80b051af3eb8"}
Sep 17 23:18:31.513952 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.513913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f2qc4" event={"ID":"bc44ba4c-4048-4f38-b017-1f1a5c2ddda3","Type":"ContainerStarted","Data":"1f9c0bbca5b43dd4dfff64d51b809c2924bb70030991f6cf44082fc91d9159fb"}
Sep 17 23:18:31.514435 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.514020    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:31.515337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.515313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq" event={"ID":"bea51e17-c521-497d-b499-84b634e06cff","Type":"ContainerStarted","Data":"9bb680b86f17253ba95c7365e4bab9ea7458bf654aab66c819dda9f5262f0b39"}
Sep 17 23:18:31.515462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.515433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:18:31.533487 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.533445    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-f2qc4" podStartSLOduration=2.365357476 podStartE2EDuration="6.533432322s" totalImagesPullingTime="4.168074846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:31.53249403 +0000 UTC m=+466.340605181" watchObservedRunningTime="2026-09-17 23:18:31.533432322 +0000 UTC m=+466.341543470"
Sep 17 23:18:31.549192 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:31.549133    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq" podStartSLOduration=3.042441048 podStartE2EDuration="6.549111093s" totalImagesPullingTime="3.506670045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:31.547782447 +0000 UTC m=+466.355893597" watchObservedRunningTime="2026-09-17 23:18:31.549111093 +0000 UTC m=+466.357222243"
Sep 17 23:18:37.522121 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:37.522090    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-f2qc4"
Sep 17 23:18:39.775806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.775773    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6f8d75b-jsx26"]
Sep 17 23:18:39.778721 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.778703    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.791518 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.791491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6f8d75b-jsx26"]
Sep 17 23:18:39.841142 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.841121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.863234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp8h\" (UniqueName: \"kubernetes.io/projected/0055cfb2-1efa-406e-8532-d706e20ba7bf-kube-api-access-rnp8h\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863306 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863306 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-trusted-ca-bundle\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863376 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-oauth-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863413 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863413 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-service-ca\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.863473 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.863419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-oauth-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.905846 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.905820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.964157 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-service-ca\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-oauth-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp8h\" (UniqueName: \"kubernetes.io/projected/0055cfb2-1efa-406e-8532-d706e20ba7bf-kube-api-access-rnp8h\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964418 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964252    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-trusted-ca-bundle\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964418 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-oauth-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.964931 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.965028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.965001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-oauth-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.965028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.964965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-service-ca\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.965185 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.965169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0055cfb2-1efa-406e-8532-d706e20ba7bf-trusted-ca-bundle\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.966577 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.966552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-oauth-config\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.966750 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.966732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0055cfb2-1efa-406e-8532-d706e20ba7bf-console-serving-cert\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:39.973845 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:39.973825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp8h\" (UniqueName: \"kubernetes.io/projected/0055cfb2-1efa-406e-8532-d706e20ba7bf-kube-api-access-rnp8h\") pod \"console-6b6f8d75b-jsx26\" (UID: \"0055cfb2-1efa-406e-8532-d706e20ba7bf\") " pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:40.088229 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:40.088182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:40.374389 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:40.374366    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6f8d75b-jsx26"]
Sep 17 23:18:40.377108 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:18:40.377081    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0055cfb2_1efa_406e_8532_d706e20ba7bf.slice/crio-7caf69cc9899cf53dbc9da0cc16ab5cd0436c96d5765114e6a25ea2278c47d7b WatchSource:0}: Error finding container 7caf69cc9899cf53dbc9da0cc16ab5cd0436c96d5765114e6a25ea2278c47d7b: Status 404 returned error can't find the container with id 7caf69cc9899cf53dbc9da0cc16ab5cd0436c96d5765114e6a25ea2278c47d7b
Sep 17 23:18:40.552258 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:40.552228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6f8d75b-jsx26" event={"ID":"0055cfb2-1efa-406e-8532-d706e20ba7bf","Type":"ContainerStarted","Data":"f33bf62e94496346d1915f3bcf0169f1b81811a02249ae35534cdd2ded09dead"}
Sep 17 23:18:40.552402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:40.552284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6f8d75b-jsx26" event={"ID":"0055cfb2-1efa-406e-8532-d706e20ba7bf","Type":"ContainerStarted","Data":"7caf69cc9899cf53dbc9da0cc16ab5cd0436c96d5765114e6a25ea2278c47d7b"}
Sep 17 23:18:40.572942 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:40.572899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6f8d75b-jsx26" podStartSLOduration=1.572884734 podStartE2EDuration="1.572884734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:40.571931939 +0000 UTC m=+475.380043089" watchObservedRunningTime="2026-09-17 23:18:40.572884734 +0000 UTC m=+475.380995881"
Sep 17 23:18:50.088795 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.088763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:50.089152 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.088818    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:50.093072 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.093051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:50.592064 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.592041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6f8d75b-jsx26"
Sep 17 23:18:50.638779 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.638750    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:18:50.749284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.749240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:50.849633 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.849562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:50.915227 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:18:50.915200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:02.521944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:02.521917    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-jvsvq"
Sep 17 23:19:15.662668 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.662616    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75d94d4b7b-p6x6p" podUID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" containerName="console" containerID="cri-o://e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e" gracePeriod=15
Sep 17 23:19:15.894568 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.894544    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:19:15.915187 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915140    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb7r2\" (UniqueName: \"kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915187 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915172    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915211    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915233    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915268    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915350 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915337    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config\") pod \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\" (UID: \"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1\") "
Sep 17 23:19:15.915682 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config" (OuterVolumeSpecName: "console-config") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:15.915746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915651    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca" (OuterVolumeSpecName: "service-ca") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:15.915746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:15.915746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.915692    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:15.917166 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.917145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2" (OuterVolumeSpecName: "kube-api-access-wb7r2") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "kube-api-access-wb7r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:15.917279 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.917264    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:19:15.917322 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:15.917298    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" (UID: "3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:19:16.016256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016231    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7r2\" (UniqueName: \"kubernetes.io/projected/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-kube-api-access-wb7r2\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016253    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016263    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016272    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-service-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016280    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016288    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-oauth-serving-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.016379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.016296    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1-console-oauth-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:19:16.693086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.693055    2559 generic.go:356] "Generic (PLEG): container finished" podID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" containerID="e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e" exitCode=0
Sep 17 23:19:16.693501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.693102    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d94d4b7b-p6x6p"
Sep 17 23:19:16.693501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.693147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d94d4b7b-p6x6p" event={"ID":"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1","Type":"ContainerDied","Data":"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"}
Sep 17 23:19:16.693501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.693196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d94d4b7b-p6x6p" event={"ID":"3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1","Type":"ContainerDied","Data":"d0171d7f5da63742ffca9655778f0f92844fcab0626c1daf162fc8af95ec446c"}
Sep 17 23:19:16.693501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.693216    2559 scope.go:117] "RemoveContainer" containerID="e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"
Sep 17 23:19:16.702077 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.702061    2559 scope.go:117] "RemoveContainer" containerID="e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"
Sep 17 23:19:16.702348 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:19:16.702314    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e\": container with ID starting with e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e not found: ID does not exist" containerID="e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"
Sep 17 23:19:16.702400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.702348    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e"} err="failed to get container status \"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e\": rpc error: code = NotFound desc = could not find container \"e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e\": container with ID starting with e14a51e9da6e16c91d68e5fad4f6f97b9bc916bf7e47aff1adea14272af1689e not found: ID does not exist"
Sep 17 23:19:16.714445 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.714424    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:19:16.717918 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:16.717897    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75d94d4b7b-p6x6p"]
Sep 17 23:19:17.647962 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:17.647930    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" path="/var/lib/kubelet/pods/3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1/volumes"
Sep 17 23:19:42.984173 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:42.984083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:43.049516 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:43.049480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.526691 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.526639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.581543 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.581450    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-bnxsq"]
Sep 17 23:19:44.581861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.581844    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" containerName="console"
Sep 17 23:19:44.581974 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.581865    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" containerName="console"
Sep 17 23:19:44.581974 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.581956    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc2793f-2b45-4d10-9b3e-de95e7f9bfa1" containerName="console"
Sep 17 23:19:44.583774 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.583756    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.586462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.586440    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 23:19:44.586914 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.586892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-fqsxg\""
Sep 17 23:19:44.587428 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.587414    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 23:19:44.596800 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.596772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bnxsq"]
Sep 17 23:19:44.619642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.619614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.726149 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.726122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.726237 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.726160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.726237 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.726208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcl9r\" (UniqueName: \"kubernetes.io/projected/78707091-83de-4ffb-ba94-d93b11ff3ad4-kube-api-access-gcl9r\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.739902 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.739875    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.826878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.826822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.826878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.826859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.827029 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.826901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcl9r\" (UniqueName: \"kubernetes.io/projected/78707091-83de-4ffb-ba94-d93b11ff3ad4-kube-api-access-gcl9r\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.827029 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:19:44.826954    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 23:19:44.827029 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:19:44.827011    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 23:19:44.827187 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:19:44.827017    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert podName:78707091-83de-4ffb-ba94-d93b11ff3ad4 nodeName:}" failed. No retries permitted until 2026-09-17 23:19:45.327001628 +0000 UTC m=+540.135112755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert") pod "odh-model-controller-7b966d7984-bnxsq" (UID: "78707091-83de-4ffb-ba94-d93b11ff3ad4") : secret "odh-model-controller-webhook-cert" not found
Sep 17 23:19:44.827187 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:19:44.827079    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs podName:78707091-83de-4ffb-ba94-d93b11ff3ad4 nodeName:}" failed. No retries permitted until 2026-09-17 23:19:45.327066143 +0000 UTC m=+540.135177270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs") pod "odh-model-controller-7b966d7984-bnxsq" (UID: "78707091-83de-4ffb-ba94-d93b11ff3ad4") : secret "odh-model-controller-metrics-tls" not found
Sep 17 23:19:44.839598 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.839575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.839890 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.839868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcl9r\" (UniqueName: \"kubernetes.io/projected/78707091-83de-4ffb-ba94-d93b11ff3ad4-kube-api-access-gcl9r\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:44.954024 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:44.953996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:45.330809 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.330788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:45.330909 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.330819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:45.332988 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.332968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-metrics-certs\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:45.333064 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.333037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78707091-83de-4ffb-ba94-d93b11ff3ad4-cert\") pod \"odh-model-controller-7b966d7984-bnxsq\" (UID: \"78707091-83de-4ffb-ba94-d93b11ff3ad4\") " pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:45.358702 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.358682    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:45.494205 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.494170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:45.794919 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.794895    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-bnxsq"]
Sep 17 23:19:45.796888 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:19:45.796863    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78707091_83de_4ffb_ba94_d93b11ff3ad4.slice/crio-145d2a407e3f5aca5e48a3ade684193b82a9150307d8168a876ff775fb40398d WatchSource:0}: Error finding container 145d2a407e3f5aca5e48a3ade684193b82a9150307d8168a876ff775fb40398d: Status 404 returned error can't find the container with id 145d2a407e3f5aca5e48a3ade684193b82a9150307d8168a876ff775fb40398d
Sep 17 23:19:45.805323 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.805301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bnxsq" event={"ID":"78707091-83de-4ffb-ba94-d93b11ff3ad4","Type":"ContainerStarted","Data":"145d2a407e3f5aca5e48a3ade684193b82a9150307d8168a876ff775fb40398d"}
Sep 17 23:19:45.862257 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.862207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:45.939878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:45.939856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:48.820198 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:48.820163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-bnxsq" event={"ID":"78707091-83de-4ffb-ba94-d93b11ff3ad4","Type":"ContainerStarted","Data":"f3cccb78e4517146c575e6e9ce9b29db400ea2b5ea6799fecd9cccdbc85d333e"}
Sep 17 23:19:48.820620 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:48.820300    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:19:48.839211 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:48.839139    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-bnxsq" podStartSLOduration=2.44263922 podStartE2EDuration="4.8391233s" totalImagesPullingTime="2.39648408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:48.836877032 +0000 UTC m=+543.644988179" watchObservedRunningTime="2026-09-17 23:19:48.8391233 +0000 UTC m=+543.647234448"
Sep 17 23:19:59.826391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:19:59.826356    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-bnxsq"
Sep 17 23:20:00.632151 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.632114    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cssbc"]
Sep 17 23:20:00.635581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.635558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.637877 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.637856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:20:00.644695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.644641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cssbc"]
Sep 17 23:20:00.745345 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.745322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklkq\" (UniqueName: \"kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.745436 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.745393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.846139 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.846110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklkq\" (UniqueName: \"kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.846512 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.846175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.846512 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.846468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.855007 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.854978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklkq\" (UniqueName: \"kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq\") pod \"s3-init-cssbc\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") " pod="kserve/s3-init-cssbc"
Sep 17 23:20:00.945415 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:00.945390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cssbc"
Sep 17 23:20:01.216557 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:01.216534    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cssbc"]
Sep 17 23:20:01.218704 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:20:01.218673    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d6854b8_62c8_4a08_a72f_600aa5b57f2c.slice/crio-c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811 WatchSource:0}: Error finding container c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811: Status 404 returned error can't find the container with id c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811
Sep 17 23:20:01.874772 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:01.874729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerStarted","Data":"c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811"}
Sep 17 23:20:04.889026 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:04.888989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerStarted","Data":"8c9ff69c2cf624fdf99e06351d8ce6c6a3ffaef3fd05a469ddbe4314236b552d"}
Sep 17 23:20:05.894149 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:05.894081    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerID="8c9ff69c2cf624fdf99e06351d8ce6c6a3ffaef3fd05a469ddbe4314236b552d" exitCode=0
Sep 17 23:20:05.894149 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:05.894132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerDied","Data":"8c9ff69c2cf624fdf99e06351d8ce6c6a3ffaef3fd05a469ddbe4314236b552d"}
Sep 17 23:20:10.917227 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:10.917194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerStarted","Data":"0e5b9cd169179822c471d2edc219c11b6af4af6acecb837acc97048ddee03ec5"}
Sep 17 23:20:10.937590 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:10.937540    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cssbc" podStartSLOduration=2.44808049 podStartE2EDuration="10.937524932s" totalImagesPullingTime="6.908670764s" totalInitContainerRuntime="1.580773678s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:10.935146759 +0000 UTC m=+565.743257909" watchObservedRunningTime="2026-09-17 23:20:10.937524932 +0000 UTC m=+565.745636081"
Sep 17 23:20:13.193628 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:13.193587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:17.698504 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:20:17.698472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:15.173229 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:15.173197    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerID="0e5b9cd169179822c471d2edc219c11b6af4af6acecb837acc97048ddee03ec5" exitCode=0
Sep 17 23:21:15.173582 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:15.173234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerDied","Data":"0e5b9cd169179822c471d2edc219c11b6af4af6acecb837acc97048ddee03ec5"}
Sep 17 23:21:16.294949 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.294926    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cssbc"
Sep 17 23:21:16.390420 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.390394    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fklkq\" (UniqueName: \"kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq\") pod \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") "
Sep 17 23:21:16.390518 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.390434    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache\") pod \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\" (UID: \"1d6854b8-62c8-4a08-a72f-600aa5b57f2c\") "
Sep 17 23:21:16.392368 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.392343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq" (OuterVolumeSpecName: "kube-api-access-fklkq") pod "1d6854b8-62c8-4a08-a72f-600aa5b57f2c" (UID: "1d6854b8-62c8-4a08-a72f-600aa5b57f2c"). InnerVolumeSpecName "kube-api-access-fklkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:16.408463 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.408441    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d6854b8-62c8-4a08-a72f-600aa5b57f2c" (UID: "1d6854b8-62c8-4a08-a72f-600aa5b57f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:21:16.491058 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.491037    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklkq\" (UniqueName: \"kubernetes.io/projected/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-kube-api-access-fklkq\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:21:16.491133 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:16.491059    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6854b8-62c8-4a08-a72f-600aa5b57f2c-model-cache\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:21:17.181542 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:17.181510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cssbc" event={"ID":"1d6854b8-62c8-4a08-a72f-600aa5b57f2c","Type":"ContainerDied","Data":"c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811"}
Sep 17 23:21:17.181542 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:17.181547    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c827d683fa512a8e1fab1ba6de3ad641acae67547b3ab97af8b44732c019a811"
Sep 17 23:21:17.181746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:17.181564    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cssbc"
Sep 17 23:21:17.565116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:17.565043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:17.662885 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:17.662856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:28.978550 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:28.978520    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.038941 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.038912    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:21:29.039209 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.039198    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerName="s3-init"
Sep 17 23:21:29.039251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.039210    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerName="s3-init"
Sep 17 23:21:29.039251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.039233    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerName="download-hf-model"
Sep 17 23:21:29.039251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.039239    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerName="download-hf-model"
Sep 17 23:21:29.039392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.039292    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6854b8-62c8-4a08-a72f-600aa5b57f2c" containerName="s3-init"
Sep 17 23:21:29.042561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.042547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.045168 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.045147    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9ls6r\""
Sep 17 23:21:29.045385 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.045367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:21:29.045471 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.045385    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 17 23:21:29.045471 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.045435    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 17 23:21:29.045629 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.045615    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:21:29.046518 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.046500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:21:29.046619 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.046580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:21:29.052520 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.052500    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:21:29.172733 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.172703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.182653 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.182619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.182774 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.182684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.182774 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.182716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.182774 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.182741    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56dw\" (UniqueName: \"kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.182774 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.182765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283208 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283431 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283431 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283431 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56dw\" (UniqueName: \"kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283431 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.283648 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.283601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.284089 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.284062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.284214 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.284193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.285956 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.285936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.294498 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.294471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56dw\" (UniqueName: \"kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw\") pod \"isvc-sklearn-graph-1-predictor-786b497f89-ntlt8\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.302077 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.302055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.339295 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.339263    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:21:29.343338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.343323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.347338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.347320    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 17 23:21:29.349161 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.349146    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 17 23:21:29.353346 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.353328    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:29.356724 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.356703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:21:29.403678 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.403191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.429434 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.429406    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:21:29.435779 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.435758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.440248 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.438155    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6ba0-kube-rbac-proxy-sar-config\""
Sep 17 23:21:29.440248 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.438382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6ba0-predictor-serving-cert\""
Sep 17 23:21:29.444290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.443885    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:21:29.486298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.486112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.486298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.486153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.486298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.486201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.486298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.486222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q425w\" (UniqueName: \"kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.486298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.486244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.499704 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.499653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.586938 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.586863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.586935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.586967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbg4\" (UniqueName: \"kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.587060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q425w\" (UniqueName: \"kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587572 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.587572 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587572 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.587572 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:21:29.587393    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 17 23:21:29.587572 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:21:29.587464    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls podName:fec938fb-b703-47d1-b349-0b4022eacb08 nodeName:}" failed. No retries permitted until 2026-09-17 23:21:30.087445242 +0000 UTC m=+644.895556377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-xjn72" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Sep 17 23:21:29.587859 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587966 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.587966 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.587943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.595132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.595058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q425w\" (UniqueName: \"kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:29.596414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.596390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.613868 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.613849    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:21:29.617570 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.617554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.619960 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.619937    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 17 23:21:29.619960 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.619956    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 17 23:21:29.628788 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.628771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:21:29.653965 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.653941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:21:29.655374 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:21:29.655353    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cd3953_a341_4906_97cc_0f9a96a74678.slice/crio-bf94eddd77fdd730becf234f3826cfe0dc8ac4f3924245656967448afd7ad363 WatchSource:0}: Error finding container bf94eddd77fdd730becf234f3826cfe0dc8ac4f3924245656967448afd7ad363: Status 404 returned error can't find the container with id bf94eddd77fdd730becf234f3826cfe0dc8ac4f3924245656967448afd7ad363
Sep 17 23:21:29.688353 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.688065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbg4\" (UniqueName: \"kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.688353 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.688138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.688353 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.688194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.688958 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.688935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.690690 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.690654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.697227 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.697133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.697370 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.697350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbg4\" (UniqueName: \"kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4\") pod \"error-404-isvc-b6ba0-predictor-6665558fb9-99hnd\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") " pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.751932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.751895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:29.789999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.789258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.789999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.789310    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.789999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.789347    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.789999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.789375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.789999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.789468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pfx\" (UniqueName: \"kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.797643 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.797503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.890831 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.890754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.890963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.890836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pfx\" (UniqueName: \"kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891016 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.890996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891066 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.891030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891116 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.891066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891187 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.891167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891351 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:21:29.891332    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 23:21:29.891417 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:21:29.891402    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls podName:ae705b50-d86a-4638-929a-7bb5b7315285 nodeName:}" failed. No retries permitted until 2026-09-17 23:21:30.391382671 +0000 UTC m=+645.199493809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls") pod "isvc-sklearn-graph-2-predictor-867659858b-x295x" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 23:21:29.891742 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.891723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.891832 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.891812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:29.902202 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:29.902176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pfx\" (UniqueName: \"kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:30.036856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.036829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:21:30.040329 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:21:30.040296    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d355e6_c535_408b_836a_266a3788443d.slice/crio-03e8e33e24c9428fd006fac2742f619fa0f1775fca68e48fe4daa1ab314e3c65 WatchSource:0}: Error finding container 03e8e33e24c9428fd006fac2742f619fa0f1775fca68e48fe4daa1ab314e3c65: Status 404 returned error can't find the container with id 03e8e33e24c9428fd006fac2742f619fa0f1775fca68e48fe4daa1ab314e3c65
Sep 17 23:21:30.059590 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.059549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:30.093040 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.093013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:30.095372 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.095351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-xjn72\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:30.236790 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.236746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerStarted","Data":"03e8e33e24c9428fd006fac2742f619fa0f1775fca68e48fe4daa1ab314e3c65"}
Sep 17 23:21:30.238473 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.238450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerStarted","Data":"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"}
Sep 17 23:21:30.238561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.238502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerStarted","Data":"bf94eddd77fdd730becf234f3826cfe0dc8ac4f3924245656967448afd7ad363"}
Sep 17 23:21:30.256820 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.256800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:21:30.330952 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.330884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:30.395862 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.395833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:30.398581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.398557    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-867659858b-x295x\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:30.531712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.531638    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:30.536104 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.536033    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:21:30.538024 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:21:30.537996    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec938fb_b703_47d1_b349_0b4022eacb08.slice/crio-baf2cd54cef22dfe594a19f39c12f6b2de23f4f0aaffe7c74cd5a7067f484e98 WatchSource:0}: Error finding container baf2cd54cef22dfe594a19f39c12f6b2de23f4f0aaffe7c74cd5a7067f484e98: Status 404 returned error can't find the container with id baf2cd54cef22dfe594a19f39c12f6b2de23f4f0aaffe7c74cd5a7067f484e98
Sep 17 23:21:30.834540 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:30.834461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:21:30.839570 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:21:30.839530    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae705b50_d86a_4638_929a_7bb5b7315285.slice/crio-b8c922ea8c2191b020b293ea321ef1e873812b8629506f79b29bd680ebd117c2 WatchSource:0}: Error finding container b8c922ea8c2191b020b293ea321ef1e873812b8629506f79b29bd680ebd117c2: Status 404 returned error can't find the container with id b8c922ea8c2191b020b293ea321ef1e873812b8629506f79b29bd680ebd117c2
Sep 17 23:21:31.248354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:31.248216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerStarted","Data":"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"}
Sep 17 23:21:31.248354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:31.248298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerStarted","Data":"b8c922ea8c2191b020b293ea321ef1e873812b8629506f79b29bd680ebd117c2"}
Sep 17 23:21:31.257782 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:31.257539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerStarted","Data":"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"}
Sep 17 23:21:31.257782 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:31.257601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerStarted","Data":"baf2cd54cef22dfe594a19f39c12f6b2de23f4f0aaffe7c74cd5a7067f484e98"}
Sep 17 23:21:35.286470 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:35.286430    2559 generic.go:356] "Generic (PLEG): container finished" podID="98cd3953-a341-4906-97cc-0f9a96a74678" containerID="bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551" exitCode=0
Sep 17 23:21:35.286924 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:35.286526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerDied","Data":"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"}
Sep 17 23:21:36.294498 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:36.294459    2559 generic.go:356] "Generic (PLEG): container finished" podID="ae705b50-d86a-4638-929a-7bb5b7315285" containerID="637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c" exitCode=0
Sep 17 23:21:36.294951 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:36.294530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerDied","Data":"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"}
Sep 17 23:21:36.296467 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:36.296441    2559 generic.go:356] "Generic (PLEG): container finished" podID="fec938fb-b703-47d1-b349-0b4022eacb08" containerID="e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575" exitCode=0
Sep 17 23:21:36.296591 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:36.296485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerDied","Data":"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"}
Sep 17 23:21:47.369115 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:47.369074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerStarted","Data":"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"}
Sep 17 23:21:47.375932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:47.375904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerStarted","Data":"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"}
Sep 17 23:21:47.381302 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:47.381275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerStarted","Data":"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"}
Sep 17 23:21:51.415592 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.415550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerStarted","Data":"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"}
Sep 17 23:21:51.416078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.415630    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:51.416078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.415680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:51.417561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.417529    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:21:51.418991 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.418966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerStarted","Data":"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"}
Sep 17 23:21:51.419151 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.419132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:51.421865 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.421839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerStarted","Data":"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"}
Sep 17 23:21:51.421963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.421947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:51.436682 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.436629    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podStartSLOduration=1.969145334 podStartE2EDuration="22.436615604s" totalImagesPullingTime="20.46747027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:51.435955039 +0000 UTC m=+666.244066190" watchObservedRunningTime="2026-09-17 23:21:51.436615604 +0000 UTC m=+666.244726754"
Sep 17 23:21:51.459731 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.459689    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podStartSLOduration=3.89151528 podStartE2EDuration="22.459647857s" totalImagesPullingTime="14.183232996s" totalInitContainerRuntime="4.384899581s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:51.45796218 +0000 UTC m=+666.266073329" watchObservedRunningTime="2026-09-17 23:21:51.459647857 +0000 UTC m=+666.267759007"
Sep 17 23:21:51.483299 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:51.483243    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podStartSLOduration=3.322575515 podStartE2EDuration="22.483230442s" totalImagesPullingTime="15.185706605s" totalInitContainerRuntime="3.974948322s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:51.48134811 +0000 UTC m=+666.289459372" watchObservedRunningTime="2026-09-17 23:21:51.483230442 +0000 UTC m=+666.291341592"
Sep 17 23:21:52.426757 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:52.426706    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:21:52.427396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:52.427146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:52.427396 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:52.427202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:52.427932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:52.427907    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:21:52.428004 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:52.427946    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:21:53.432582 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:53.432541    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:21:53.433017 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:53.432677    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:21:57.431424 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:57.431394    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:21:57.432048 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:57.432022    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:21:58.438211 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:58.438111    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:21:58.439595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:58.439300    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:21:58.440589 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:58.440563    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:21:58.441174 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:21:58.441149    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:07.431923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:07.431886    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:22:07.507248 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:07.507214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerStarted","Data":"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"}
Sep 17 23:22:07.507452 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:07.507306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerStarted","Data":"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"}
Sep 17 23:22:07.507515 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:07.507498    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:22:07.531486 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:07.531429    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podStartSLOduration=3.4558090630000002 podStartE2EDuration="38.531414258s" totalImagesPullingTime="30.717999202s" totalInitContainerRuntime="4.357605993s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:07.529227218 +0000 UTC m=+682.337338370" watchObservedRunningTime="2026-09-17 23:22:07.531414258 +0000 UTC m=+682.339525405"
Sep 17 23:22:08.440220 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:08.440183    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:08.441409 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:08.441374    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:08.512631 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:08.512600    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:22:08.513742 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:08.513719    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:09.516822 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:09.516782    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:14.521064 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:14.521031    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:22:14.521542 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:14.521516    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:15.182925 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:15.182882    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:17.432599 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:17.432551    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:22:18.440314 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:18.440273    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:18.441501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:18.441478    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:24.522110 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:24.522071    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:27.432410 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:27.432367    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:22:28.439896 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:28.439856    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:28.441083 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:28.441057    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:34.521620 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:34.521576    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:37.432208 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:37.432168    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:22:38.439487 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:38.439407    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:38.441684 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:38.441644    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:44.522034 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:44.521994    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:47.432943 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:47.432916    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:22:48.439858 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:48.439818    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:48.441014 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:48.440989    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:22:48.956273 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:48.956239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:49.719836 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:49.719807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:54.521490 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:54.521452    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:22:58.439740 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:58.439694    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:22:58.441985 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:22:58.441951    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:23:03.109382 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.109340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.284453 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.284400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.292039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.292000    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:23:03.292462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.292261    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" containerID="cri-o://29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b" gracePeriod=30
Sep 17 23:23:03.292462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.292318    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kube-rbac-proxy" containerID="cri-o://7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104" gracePeriod=30
Sep 17 23:23:03.388783 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.388614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.492700 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.492588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.542980 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.542942    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:23:03.546259 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.546230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.553202 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.552952    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-11096-predictor-serving-cert\""
Sep 17 23:23:03.553455 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.553367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-11096-kube-rbac-proxy-sar-config\""
Sep 17 23:23:03.557589 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.557566    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:23:03.569314 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.569281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.569511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.569364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.569511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.569389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tjd\" (UniqueName: \"kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.590684 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.590548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.644241 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.644140    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:23:03.670490 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.670448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.670490 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.670487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tjd\" (UniqueName: \"kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.670780 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.670576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.671233 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.671209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.673061 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.673041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.680281 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.680250    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tjd\" (UniqueName: \"kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd\") pod \"error-404-isvc-11096-predictor-5d7bddccd5-6nvn4\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") " pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:03.688907 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.688828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.763425 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.763390    2559 generic.go:356] "Generic (PLEG): container finished" podID="d8d355e6-c535-408b-836a-266a3788443d" containerID="7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104" exitCode=2
Sep 17 23:23:03.763603 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.763468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerDied","Data":"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"}
Sep 17 23:23:03.810464 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.810427    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:03.861073 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:03.861033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:04.138786 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.138760    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:23:04.140723 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:23:04.140684    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb23a376_b313_42e2_82cb_2ae293396b0f.slice/crio-fab3667a476d50eae51174e077b50f903d4f22a0c91a11175ab91c5e4f3a632e WatchSource:0}: Error finding container fab3667a476d50eae51174e077b50f903d4f22a0c91a11175ab91c5e4f3a632e: Status 404 returned error can't find the container with id fab3667a476d50eae51174e077b50f903d4f22a0c91a11175ab91c5e4f3a632e
Sep 17 23:23:04.521616 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.521576    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:23:04.769886 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.769844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerStarted","Data":"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"}
Sep 17 23:23:04.770062 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.769916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerStarted","Data":"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"}
Sep 17 23:23:04.770062 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.769946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerStarted","Data":"fab3667a476d50eae51174e077b50f903d4f22a0c91a11175ab91c5e4f3a632e"}
Sep 17 23:23:04.770062 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.770055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:04.792099 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:04.791972    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podStartSLOduration=1.791950921 podStartE2EDuration="1.791950921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:04.791306954 +0000 UTC m=+739.599418123" watchObservedRunningTime="2026-09-17 23:23:04.791950921 +0000 UTC m=+739.600062071"
Sep 17 23:23:05.774883 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:05.774850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:05.776294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:05.776267    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:06.779866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:06.779831    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:07.439226 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.439205    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:23:07.503550 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.503523    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls\") pod \"d8d355e6-c535-408b-836a-266a3788443d\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") "
Sep 17 23:23:07.503701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.503559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config\") pod \"d8d355e6-c535-408b-836a-266a3788443d\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") "
Sep 17 23:23:07.503701 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.503587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqbg4\" (UniqueName: \"kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4\") pod \"d8d355e6-c535-408b-836a-266a3788443d\" (UID: \"d8d355e6-c535-408b-836a-266a3788443d\") "
Sep 17 23:23:07.504009 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.503978    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8d355e6-c535-408b-836a-266a3788443d" (UID: "d8d355e6-c535-408b-836a-266a3788443d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:07.505696 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.505669    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4" (OuterVolumeSpecName: "kube-api-access-rqbg4") pod "d8d355e6-c535-408b-836a-266a3788443d" (UID: "d8d355e6-c535-408b-836a-266a3788443d"). InnerVolumeSpecName "kube-api-access-rqbg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:07.505696 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.505683    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8d355e6-c535-408b-836a-266a3788443d" (UID: "d8d355e6-c535-408b-836a-266a3788443d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:07.605057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.604997    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8d355e6-c535-408b-836a-266a3788443d-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:07.605057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.605019    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8d355e6-c535-408b-836a-266a3788443d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:07.605057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.605028    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqbg4\" (UniqueName: \"kubernetes.io/projected/d8d355e6-c535-408b-836a-266a3788443d-kube-api-access-rqbg4\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:07.787150 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.787111    2559 generic.go:356] "Generic (PLEG): container finished" podID="d8d355e6-c535-408b-836a-266a3788443d" containerID="29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b" exitCode=0
Sep 17 23:23:07.787588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.787158    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"
Sep 17 23:23:07.787588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.787168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerDied","Data":"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"}
Sep 17 23:23:07.787588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.787233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" event={"ID":"d8d355e6-c535-408b-836a-266a3788443d","Type":"ContainerDied","Data":"03e8e33e24c9428fd006fac2742f619fa0f1775fca68e48fe4daa1ab314e3c65"}
Sep 17 23:23:07.787588 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.787271    2559 scope.go:117] "RemoveContainer" containerID="7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"
Sep 17 23:23:07.796275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.796259    2559 scope.go:117] "RemoveContainer" containerID="29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"
Sep 17 23:23:07.804255 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.804236    2559 scope.go:117] "RemoveContainer" containerID="7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"
Sep 17 23:23:07.804505 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:07.804482    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104\": container with ID starting with 7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104 not found: ID does not exist" containerID="7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"
Sep 17 23:23:07.804568 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.804517    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104"} err="failed to get container status \"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104\": rpc error: code = NotFound desc = could not find container \"7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104\": container with ID starting with 7339ee95f954d6a623daeedf044c9aa23c550a6d4bd8e6fc50890162150ad104 not found: ID does not exist"
Sep 17 23:23:07.804568 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.804534    2559 scope.go:117] "RemoveContainer" containerID="29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"
Sep 17 23:23:07.804787 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:07.804766    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b\": container with ID starting with 29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b not found: ID does not exist" containerID="29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"
Sep 17 23:23:07.804858 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.804792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b"} err="failed to get container status \"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b\": rpc error: code = NotFound desc = could not find container \"29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b\": container with ID starting with 29e1500c8a4928cfdbfe499e68ccc84be70701ec32b369d434e9f32d06dc154b not found: ID does not exist"
Sep 17 23:23:07.805133 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.805114    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:23:07.809039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:07.809019    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd"]
Sep 17 23:23:08.427132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:08.427088    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:23:08.432629 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:08.432595    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6ba0-predictor-6665558fb9-99hnd" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: i/o timeout"
Sep 17 23:23:08.439807 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:08.439789    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:23:09.649498 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:09.649459    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d355e6-c535-408b-836a-266a3788443d" path="/var/lib/kubelet/pods/d8d355e6-c535-408b-836a-266a3788443d/volumes"
Sep 17 23:23:11.786480 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:11.786450    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:11.787070 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:11.787041    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:13.649475 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:13.649443    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:23:14.522107 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:14.522067    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:23:21.787928 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:21.787888    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:24.522235 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:24.522199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:23:25.074586 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:25.074543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:25.134342 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:25.134308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:30.438776 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:30.438728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:30.506525 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:30.506491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:31.787383 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:31.787338    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:38.994141 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:38.994105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:39.220126 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:39.220090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.787223 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:41.787177    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:23:49.647794 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.647762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:49.779590 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.779552    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:23:49.779855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.779830    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" containerID="cri-o://ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc" gracePeriod=30
Sep 17 23:23:49.779948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.779860    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kube-rbac-proxy" containerID="cri-o://a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb" gracePeriod=30
Sep 17 23:23:49.842642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.842613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:49.870073 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.870049    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:23:49.870287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.870259    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container" containerID="cri-o://c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86" gracePeriod=30
Sep 17 23:23:49.870407 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.870318    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kube-rbac-proxy" containerID="cri-o://1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d" gracePeriod=30
Sep 17 23:23:49.910864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.910805    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:23:49.911455 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.911397    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container" containerID="cri-o://46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c" gracePeriod=30
Sep 17 23:23:49.911806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.911771    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kube-rbac-proxy" containerID="cri-o://1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b" gracePeriod=30
Sep 17 23:23:49.997284 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.997251    2559 generic.go:356] "Generic (PLEG): container finished" podID="fec938fb-b703-47d1-b349-0b4022eacb08" containerID="1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d" exitCode=2
Sep 17 23:23:49.997418 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.997276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerDied","Data":"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"}
Sep 17 23:23:49.999710 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.999686    2559 generic.go:356] "Generic (PLEG): container finished" podID="98cd3953-a341-4906-97cc-0f9a96a74678" containerID="a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb" exitCode=2
Sep 17 23:23:49.999804 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:49.999712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerDied","Data":"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"}
Sep 17 23:23:50.049810 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.049787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.154810 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.154773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.212374 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212330    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:23:50.212771 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212757    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container"
Sep 17 23:23:50.212823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212774    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container"
Sep 17 23:23:50.212823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212788    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kube-rbac-proxy"
Sep 17 23:23:50.212823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212793    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kube-rbac-proxy"
Sep 17 23:23:50.212921 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212846    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kserve-container"
Sep 17 23:23:50.212921 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.212857    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d355e6-c535-408b-836a-266a3788443d" containerName="kube-rbac-proxy"
Sep 17 23:23:50.215861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.215847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.219397 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.219374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2026c-predictor-serving-cert\""
Sep 17 23:23:50.219996 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.219978    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2026c-kube-rbac-proxy-sar-config\""
Sep 17 23:23:50.233886 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.233863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:23:50.288915 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.288881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.327275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.327247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.327375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.327289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.327467 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.327446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967sb\" (UniqueName: \"kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.419880 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.419851    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.427878 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.427853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967sb\" (UniqueName: \"kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.427986 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.427905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.427986 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.427941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.428533 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.428508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.430267 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.430250    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.439922 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.439901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-967sb\" (UniqueName: \"kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb\") pod \"error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") " pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.526793 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.526747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:50.810054 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:23:50.810021    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6ad9c8_ca76_4742_b02c_44393a72e6b6.slice/crio-de5e8a3f808468fdbf4543c6fbcb14c19538b4a08e3fbde21039dee3a12e93bb WatchSource:0}: Error finding container de5e8a3f808468fdbf4543c6fbcb14c19538b4a08e3fbde21039dee3a12e93bb: Status 404 returned error can't find the container with id de5e8a3f808468fdbf4543c6fbcb14c19538b4a08e3fbde21039dee3a12e93bb
Sep 17 23:23:50.812746 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.812719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:23:50.929322 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:50.929295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:51.006309 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.006280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerStarted","Data":"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"}
Sep 17 23:23:51.006451 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.006343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerStarted","Data":"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"}
Sep 17 23:23:51.006451 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.006381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerStarted","Data":"de5e8a3f808468fdbf4543c6fbcb14c19538b4a08e3fbde21039dee3a12e93bb"}
Sep 17 23:23:51.006451 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.006404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:51.008423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.008398    2559 generic.go:356] "Generic (PLEG): container finished" podID="ae705b50-d86a-4638-929a-7bb5b7315285" containerID="1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b" exitCode=2
Sep 17 23:23:51.008516 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.008460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerDied","Data":"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"}
Sep 17 23:23:51.029413 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.029368    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podStartSLOduration=1.029353521 podStartE2EDuration="1.029353521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:51.025979473 +0000 UTC m=+785.834090633" watchObservedRunningTime="2026-09-17 23:23:51.029353521 +0000 UTC m=+785.837464670"
Sep 17 23:23:51.787873 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:51.787838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:23:52.013002 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:52.012972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:52.014130 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:52.014107    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:23:53.020003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:53.019962    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:23:53.433020 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:53.432975    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 23:23:53.433209 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:53.432986    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 23:23:53.644224 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:53.644177    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 23:23:53.922955 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:53.922930    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:23:54.026477 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.026399    2559 generic.go:356] "Generic (PLEG): container finished" podID="fec938fb-b703-47d1-b349-0b4022eacb08" containerID="c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86" exitCode=0
Sep 17 23:23:54.026477 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.026450    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"
Sep 17 23:23:54.026923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.026447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerDied","Data":"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"}
Sep 17 23:23:54.026923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.026575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72" event={"ID":"fec938fb-b703-47d1-b349-0b4022eacb08","Type":"ContainerDied","Data":"baf2cd54cef22dfe594a19f39c12f6b2de23f4f0aaffe7c74cd5a7067f484e98"}
Sep 17 23:23:54.026923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.026603    2559 scope.go:117] "RemoveContainer" containerID="1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"
Sep 17 23:23:54.034560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.034547    2559 scope.go:117] "RemoveContainer" containerID="c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"
Sep 17 23:23:54.041781 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.041764    2559 scope.go:117] "RemoveContainer" containerID="e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"
Sep 17 23:23:54.048854 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.048840    2559 scope.go:117] "RemoveContainer" containerID="1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"
Sep 17 23:23:54.049107 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:54.049081    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d\": container with ID starting with 1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d not found: ID does not exist" containerID="1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"
Sep 17 23:23:54.049161 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.049108    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d"} err="failed to get container status \"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d\": rpc error: code = NotFound desc = could not find container \"1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d\": container with ID starting with 1793c8f054659933c7fb3b0d2762896b587e77e15f25e5b26aa0f39a0c238e4d not found: ID does not exist"
Sep 17 23:23:54.049161 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.049125    2559 scope.go:117] "RemoveContainer" containerID="c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"
Sep 17 23:23:54.049393 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:54.049367    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86\": container with ID starting with c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86 not found: ID does not exist" containerID="c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"
Sep 17 23:23:54.049468 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.049396    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86"} err="failed to get container status \"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86\": rpc error: code = NotFound desc = could not find container \"c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86\": container with ID starting with c5b16a8af28e2556c921fc98e18a0433fd3e46de3192d77ec7ba20cef2a2dd86 not found: ID does not exist"
Sep 17 23:23:54.049468 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.049421    2559 scope.go:117] "RemoveContainer" containerID="e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"
Sep 17 23:23:54.049708 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:54.049687    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575\": container with ID starting with e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575 not found: ID does not exist" containerID="e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"
Sep 17 23:23:54.049755 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.049719    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575"} err="failed to get container status \"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575\": rpc error: code = NotFound desc = could not find container \"e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575\": container with ID starting with e2846b272019f486d4e90a05072e289d88d2be3234eca188437371e82ae79575 not found: ID does not exist"
Sep 17 23:23:54.058561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.058543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location\") pod \"fec938fb-b703-47d1-b349-0b4022eacb08\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") "
Sep 17 23:23:54.058640 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.058596    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") pod \"fec938fb-b703-47d1-b349-0b4022eacb08\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") "
Sep 17 23:23:54.058739 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.058724    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q425w\" (UniqueName: \"kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w\") pod \"fec938fb-b703-47d1-b349-0b4022eacb08\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") "
Sep 17 23:23:54.058793 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.058756    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert\") pod \"fec938fb-b703-47d1-b349-0b4022eacb08\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") "
Sep 17 23:23:54.059171 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059126    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fec938fb-b703-47d1-b349-0b4022eacb08" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:54.059292 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059200    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config\") pod \"fec938fb-b703-47d1-b349-0b4022eacb08\" (UID: \"fec938fb-b703-47d1-b349-0b4022eacb08\") "
Sep 17 23:23:54.059861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fec938fb-b703-47d1-b349-0b4022eacb08" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.059861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059808    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fec938fb-b703-47d1-b349-0b4022eacb08" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.060039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059888    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-cabundle-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.060039 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.059914    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fec938fb-b703-47d1-b349-0b4022eacb08-kserve-provision-location\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.066383 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.066361    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fec938fb-b703-47d1-b349-0b4022eacb08" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:54.066617 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.066595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w" (OuterVolumeSpecName: "kube-api-access-q425w") pod "fec938fb-b703-47d1-b349-0b4022eacb08" (UID: "fec938fb-b703-47d1-b349-0b4022eacb08"). InnerVolumeSpecName "kube-api-access-q425w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:54.160946 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.160918    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q425w\" (UniqueName: \"kubernetes.io/projected/fec938fb-b703-47d1-b349-0b4022eacb08-kube-api-access-q425w\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.160946 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.160941    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fec938fb-b703-47d1-b349-0b4022eacb08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.161091 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.160951    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fec938fb-b703-47d1-b349-0b4022eacb08-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.349409 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.349383    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:23:54.356736 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.356715    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-xjn72"]
Sep 17 23:23:54.739470 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.739445    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:23:54.741946 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.741927    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:23:54.867320 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867247    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config\") pod \"98cd3953-a341-4906-97cc-0f9a96a74678\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") "
Sep 17 23:23:54.867320 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location\") pod \"98cd3953-a341-4906-97cc-0f9a96a74678\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") "
Sep 17 23:23:54.867519 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert\") pod \"98cd3953-a341-4906-97cc-0f9a96a74678\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") "
Sep 17 23:23:54.867519 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4pfx\" (UniqueName: \"kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx\") pod \"ae705b50-d86a-4638-929a-7bb5b7315285\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") "
Sep 17 23:23:54.867519 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867489    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config\") pod \"ae705b50-d86a-4638-929a-7bb5b7315285\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") "
Sep 17 23:23:54.867519 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56dw\" (UniqueName: \"kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw\") pod \"98cd3953-a341-4906-97cc-0f9a96a74678\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") "
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location\") pod \"ae705b50-d86a-4638-929a-7bb5b7315285\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") "
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867575    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls\") pod \"98cd3953-a341-4906-97cc-0f9a96a74678\" (UID: \"98cd3953-a341-4906-97cc-0f9a96a74678\") "
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") pod \"ae705b50-d86a-4638-929a-7bb5b7315285\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") "
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert\") pod \"ae705b50-d86a-4638-929a-7bb5b7315285\" (UID: \"ae705b50-d86a-4638-929a-7bb5b7315285\") "
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98cd3953-a341-4906-97cc-0f9a96a74678" (UID: "98cd3953-a341-4906-97cc-0f9a96a74678"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:54.867761 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867737    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98cd3953-a341-4906-97cc-0f9a96a74678" (UID: "98cd3953-a341-4906-97cc-0f9a96a74678"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.868127 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867804    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98cd3953-a341-4906-97cc-0f9a96a74678" (UID: "98cd3953-a341-4906-97cc-0f9a96a74678"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.868127 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.867874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae705b50-d86a-4638-929a-7bb5b7315285" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.868127 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae705b50-d86a-4638-929a-7bb5b7315285" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:54.868127 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae705b50-d86a-4638-929a-7bb5b7315285" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:54.868127 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868119    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.868375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868139    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98cd3953-a341-4906-97cc-0f9a96a74678-kserve-provision-location\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.868375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868154    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98cd3953-a341-4906-97cc-0f9a96a74678-cabundle-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.868375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868167    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.868375 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.868181    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae705b50-d86a-4638-929a-7bb5b7315285-kserve-provision-location\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.869803 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.869776    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw" (OuterVolumeSpecName: "kube-api-access-k56dw") pod "98cd3953-a341-4906-97cc-0f9a96a74678" (UID: "98cd3953-a341-4906-97cc-0f9a96a74678"). InnerVolumeSpecName "kube-api-access-k56dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:54.869898 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.869825    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx" (OuterVolumeSpecName: "kube-api-access-d4pfx") pod "ae705b50-d86a-4638-929a-7bb5b7315285" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285"). InnerVolumeSpecName "kube-api-access-d4pfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:54.869898 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.869838    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae705b50-d86a-4638-929a-7bb5b7315285" (UID: "ae705b50-d86a-4638-929a-7bb5b7315285"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:54.869898 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.869847    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98cd3953-a341-4906-97cc-0f9a96a74678" (UID: "98cd3953-a341-4906-97cc-0f9a96a74678"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:54.968951 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.968927    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pfx\" (UniqueName: \"kubernetes.io/projected/ae705b50-d86a-4638-929a-7bb5b7315285-kube-api-access-d4pfx\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.968951 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.968948    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56dw\" (UniqueName: \"kubernetes.io/projected/98cd3953-a341-4906-97cc-0f9a96a74678-kube-api-access-k56dw\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.969086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.968957    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98cd3953-a341-4906-97cc-0f9a96a74678-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.969086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.968967    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae705b50-d86a-4638-929a-7bb5b7315285-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:54.969086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:54.968975    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae705b50-d86a-4638-929a-7bb5b7315285-cabundle-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:23:55.034672 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.034639    2559 generic.go:356] "Generic (PLEG): container finished" podID="98cd3953-a341-4906-97cc-0f9a96a74678" containerID="ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc" exitCode=0
Sep 17 23:23:55.034989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.034704    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"
Sep 17 23:23:55.034989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.034723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerDied","Data":"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"}
Sep 17 23:23:55.034989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.034796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8" event={"ID":"98cd3953-a341-4906-97cc-0f9a96a74678","Type":"ContainerDied","Data":"bf94eddd77fdd730becf234f3826cfe0dc8ac4f3924245656967448afd7ad363"}
Sep 17 23:23:55.034989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.034839    2559 scope.go:117] "RemoveContainer" containerID="a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"
Sep 17 23:23:55.036803 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.036779    2559 generic.go:356] "Generic (PLEG): container finished" podID="ae705b50-d86a-4638-929a-7bb5b7315285" containerID="46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c" exitCode=0
Sep 17 23:23:55.036891 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.036828    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"
Sep 17 23:23:55.036891 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.036855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerDied","Data":"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"}
Sep 17 23:23:55.036969 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.036912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x" event={"ID":"ae705b50-d86a-4638-929a-7bb5b7315285","Type":"ContainerDied","Data":"b8c922ea8c2191b020b293ea321ef1e873812b8629506f79b29bd680ebd117c2"}
Sep 17 23:23:55.044036 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.044017    2559 scope.go:117] "RemoveContainer" containerID="ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"
Sep 17 23:23:55.051654 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.051639    2559 scope.go:117] "RemoveContainer" containerID="bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"
Sep 17 23:23:55.059008 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.058993    2559 scope.go:117] "RemoveContainer" containerID="a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"
Sep 17 23:23:55.059248 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.059233    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb\": container with ID starting with a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb not found: ID does not exist" containerID="a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"
Sep 17 23:23:55.059307 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059254    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb"} err="failed to get container status \"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb\": rpc error: code = NotFound desc = could not find container \"a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb\": container with ID starting with a9a06a366640209c642bf3cb3f8e0c1f417d3147cf159e6f653c40472ce595fb not found: ID does not exist"
Sep 17 23:23:55.059307 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059270    2559 scope.go:117] "RemoveContainer" containerID="ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"
Sep 17 23:23:55.059518 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.059503    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc\": container with ID starting with ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc not found: ID does not exist" containerID="ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"
Sep 17 23:23:55.059569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059522    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc"} err="failed to get container status \"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc\": rpc error: code = NotFound desc = could not find container \"ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc\": container with ID starting with ae1343d7bcde4e979e953950de3a3dd8ec70447c365e87614a322556c4f854bc not found: ID does not exist"
Sep 17 23:23:55.059569 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059534    2559 scope.go:117] "RemoveContainer" containerID="bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"
Sep 17 23:23:55.059796 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.059780    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551\": container with ID starting with bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551 not found: ID does not exist" containerID="bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"
Sep 17 23:23:55.059855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059797    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:23:55.059855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059798    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551"} err="failed to get container status \"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551\": rpc error: code = NotFound desc = could not find container \"bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551\": container with ID starting with bee2bdc403edd7d973bd397caa3b638c0ec1a8cfb15c64b27dbdcaed03047551 not found: ID does not exist"
Sep 17 23:23:55.059855 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.059832    2559 scope.go:117] "RemoveContainer" containerID="1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"
Sep 17 23:23:55.065084 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.065063    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-786b497f89-ntlt8"]
Sep 17 23:23:55.067294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.067273    2559 scope.go:117] "RemoveContainer" containerID="46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"
Sep 17 23:23:55.074776 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.074760    2559 scope.go:117] "RemoveContainer" containerID="637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"
Sep 17 23:23:55.075948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.075932    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:23:55.079907 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.079887    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-867659858b-x295x"]
Sep 17 23:23:55.081989 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.081971    2559 scope.go:117] "RemoveContainer" containerID="1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"
Sep 17 23:23:55.082250 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.082227    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b\": container with ID starting with 1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b not found: ID does not exist" containerID="1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"
Sep 17 23:23:55.082291 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.082259    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b"} err="failed to get container status \"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b\": rpc error: code = NotFound desc = could not find container \"1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b\": container with ID starting with 1630a22997c1787d88ff519cdeeab07a9b81ce7abb3ec291461a9e851f36ab6b not found: ID does not exist"
Sep 17 23:23:55.082291 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.082280    2559 scope.go:117] "RemoveContainer" containerID="46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"
Sep 17 23:23:55.082546 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.082529    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c\": container with ID starting with 46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c not found: ID does not exist" containerID="46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"
Sep 17 23:23:55.082616 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.082556    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c"} err="failed to get container status \"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c\": rpc error: code = NotFound desc = could not find container \"46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c\": container with ID starting with 46193fa34e8d17d2b217a1bef337a1f47400d6ac8320400c16ece8eb59e7696c not found: ID does not exist"
Sep 17 23:23:55.082616 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.082576    2559 scope.go:117] "RemoveContainer" containerID="637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"
Sep 17 23:23:55.082856 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:23:55.082838    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c\": container with ID starting with 637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c not found: ID does not exist" containerID="637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"
Sep 17 23:23:55.082899 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.082861    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c"} err="failed to get container status \"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c\": rpc error: code = NotFound desc = could not find container \"637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c\": container with ID starting with 637dc6756a81a9636f6be7fe8b25c50fac5bb6947158b6317e8ec18e04bac68c not found: ID does not exist"
Sep 17 23:23:55.650098 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.650062    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" path="/var/lib/kubelet/pods/98cd3953-a341-4906-97cc-0f9a96a74678/volumes"
Sep 17 23:23:55.650623 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.650600    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" path="/var/lib/kubelet/pods/ae705b50-d86a-4638-929a-7bb5b7315285/volumes"
Sep 17 23:23:55.651477 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:55.651453    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" path="/var/lib/kubelet/pods/fec938fb-b703-47d1-b349-0b4022eacb08/volumes"
Sep 17 23:23:58.024326 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:58.024297    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:23:58.024708 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:23:58.024680    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:24:03.432294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:03.432262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:03.762128 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:03.762046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:08.025120 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:08.025079    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:24:18.024778 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:18.024740    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:24:28.024978 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:28.024938    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:24:38.025496 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:38.025468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:24:49.847480 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:49.847448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:49.957919 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:24:49.957889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:53.242210 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:26:53.242169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:39.188183 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:27:39.188143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:09.582940 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:30:09.582824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:15.951644 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:30:15.951594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:16.009934 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:30:16.009901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.068449 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.068413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.167489 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.167459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.175648 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.175619    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:32:18.175999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.175952    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container" containerID="cri-o://ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08" gracePeriod=30
Sep 17 23:32:18.176081 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.176007    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kube-rbac-proxy" containerID="cri-o://cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83" gracePeriod=30
Sep 17 23:32:18.263545 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.263505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.353252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.353185    2559 generic.go:356] "Generic (PLEG): container finished" podID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerID="cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83" exitCode=2
Sep 17 23:32:18.353354 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.353258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerDied","Data":"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"}
Sep 17 23:32:18.356380 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.356353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.361102 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361082    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:32:18.361428 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361415    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361430    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361445    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="storage-initializer"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361451    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="storage-initializer"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361459    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="storage-initializer"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361466    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="storage-initializer"
Sep 17 23:32:18.361476 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361474    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361480    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361488    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="storage-initializer"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361493    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="storage-initializer"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361499    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361504    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361525    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361530    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361536    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361547    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361551    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361605    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361613    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361618    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec938fb-b703-47d1-b349-0b4022eacb08" containerName="kube-rbac-proxy"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361624    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae705b50-d86a-4638-929a-7bb5b7315285" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361630    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kserve-container"
Sep 17 23:32:18.361712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.361635    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cd3953-a341-4906-97cc-0f9a96a74678" containerName="kube-rbac-proxy"
Sep 17 23:32:18.364561 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.364545    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.366998 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.366978    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e634-kube-rbac-proxy-sar-config\""
Sep 17 23:32:18.367225 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.367206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3e634-predictor-serving-cert\""
Sep 17 23:32:18.371077 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.371021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.371077 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.371056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5676\" (UniqueName: \"kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.371199 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.371110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.376597 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.376575    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:32:18.422030 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.422008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.471835 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.471809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.471944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.471839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5676\" (UniqueName: \"kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.471944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.471868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.472060 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:32:18.471954    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3e634-predictor-serving-cert: secret "error-404-isvc-3e634-predictor-serving-cert" not found
Sep 17 23:32:18.472060 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:32:18.472016    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls podName:4a19abef-acb7-4683-9c35-7563bb6cb8f4 nodeName:}" failed. No retries permitted until 2026-09-17 23:32:18.971997238 +0000 UTC m=+1293.780108381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls") pod "error-404-isvc-3e634-predictor-78db59dc7-blq7n" (UID: "4a19abef-acb7-4683-9c35-7563bb6cb8f4") : secret "error-404-isvc-3e634-predictor-serving-cert" not found
Sep 17 23:32:18.472421 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.472403    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.482486 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.482458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5676\" (UniqueName: \"kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.611930 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.611857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.712943 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.712920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.975063 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.975031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:18.977488 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:18.977467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") pod \"error-404-isvc-3e634-predictor-78db59dc7-blq7n\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") " pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:19.275645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:19.275558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:19.557733 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:19.557707    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:32:19.560379 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:32:19.560344    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a19abef_acb7_4683_9c35_7563bb6cb8f4.slice/crio-4ece32449b1e560bb8c3dca1adfb0ff8bc5bf3072a9e1759e218ac2ec875d8dd WatchSource:0}: Error finding container 4ece32449b1e560bb8c3dca1adfb0ff8bc5bf3072a9e1759e218ac2ec875d8dd: Status 404 returned error can't find the container with id 4ece32449b1e560bb8c3dca1adfb0ff8bc5bf3072a9e1759e218ac2ec875d8dd
Sep 17 23:32:20.365069 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.365035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerStarted","Data":"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"}
Sep 17 23:32:20.365437 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.365094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerStarted","Data":"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"}
Sep 17 23:32:20.365437 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.365117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerStarted","Data":"4ece32449b1e560bb8c3dca1adfb0ff8bc5bf3072a9e1759e218ac2ec875d8dd"}
Sep 17 23:32:20.365437 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.365139    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:20.384503 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.384446    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podStartSLOduration=2.384426849 podStartE2EDuration="2.384426849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:20.383862051 +0000 UTC m=+1295.191973202" watchObservedRunningTime="2026-09-17 23:32:20.384426849 +0000 UTC m=+1295.192537999"
Sep 17 23:32:20.918093 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.918068    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:32:20.993239 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.993186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls\") pod \"fb23a376-b313-42e2-82cb-2ae293396b0f\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") "
Sep 17 23:32:20.993239 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.993221    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92tjd\" (UniqueName: \"kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd\") pod \"fb23a376-b313-42e2-82cb-2ae293396b0f\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") "
Sep 17 23:32:20.993422 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.993241    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config\") pod \"fb23a376-b313-42e2-82cb-2ae293396b0f\" (UID: \"fb23a376-b313-42e2-82cb-2ae293396b0f\") "
Sep 17 23:32:20.993606 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.993584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb23a376-b313-42e2-82cb-2ae293396b0f" (UID: "fb23a376-b313-42e2-82cb-2ae293396b0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:20.995150 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.995123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd" (OuterVolumeSpecName: "kube-api-access-92tjd") pod "fb23a376-b313-42e2-82cb-2ae293396b0f" (UID: "fb23a376-b313-42e2-82cb-2ae293396b0f"). InnerVolumeSpecName "kube-api-access-92tjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:20.995150 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:20.995142    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb23a376-b313-42e2-82cb-2ae293396b0f" (UID: "fb23a376-b313-42e2-82cb-2ae293396b0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:21.093861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.093835    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb23a376-b313-42e2-82cb-2ae293396b0f-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.093861 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.093860    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tjd\" (UniqueName: \"kubernetes.io/projected/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-api-access-92tjd\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.093992 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.093869    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb23a376-b313-42e2-82cb-2ae293396b0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.371171 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371103    2559 generic.go:356] "Generic (PLEG): container finished" podID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerID="ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08" exitCode=0
Sep 17 23:32:21.371511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371170    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"
Sep 17 23:32:21.371511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerDied","Data":"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"}
Sep 17 23:32:21.371511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4" event={"ID":"fb23a376-b313-42e2-82cb-2ae293396b0f","Type":"ContainerDied","Data":"fab3667a476d50eae51174e077b50f903d4f22a0c91a11175ab91c5e4f3a632e"}
Sep 17 23:32:21.371511 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371327    2559 scope.go:117] "RemoveContainer" containerID="cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"
Sep 17 23:32:21.371782 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.371762    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:21.372787 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.372763    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:21.380244 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.380227    2559 scope.go:117] "RemoveContainer" containerID="ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"
Sep 17 23:32:21.388250 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.388232    2559 scope.go:117] "RemoveContainer" containerID="cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"
Sep 17 23:32:21.388497 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:32:21.388479    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83\": container with ID starting with cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83 not found: ID does not exist" containerID="cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"
Sep 17 23:32:21.388583 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.388502    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83"} err="failed to get container status \"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83\": rpc error: code = NotFound desc = could not find container \"cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83\": container with ID starting with cdb0c4fc76a84b52d1016e9018024fc6c4ae560d5c9d39145c0c6f4002f49d83 not found: ID does not exist"
Sep 17 23:32:21.388583 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.388516    2559 scope.go:117] "RemoveContainer" containerID="ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"
Sep 17 23:32:21.388805 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:32:21.388787    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08\": container with ID starting with ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08 not found: ID does not exist" containerID="ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"
Sep 17 23:32:21.388872 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.388806    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08"} err="failed to get container status \"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08\": rpc error: code = NotFound desc = could not find container \"ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08\": container with ID starting with ae3a76391afb9904df41f812333334ff2194729ba23a2df960c35d055653ab08 not found: ID does not exist"
Sep 17 23:32:21.393608 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.393585    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:32:21.396856 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.396838    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-11096-predictor-5d7bddccd5-6nvn4"]
Sep 17 23:32:21.648434 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:21.648361    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" path="/var/lib/kubelet/pods/fb23a376-b313-42e2-82cb-2ae293396b0f/volumes"
Sep 17 23:32:22.378156 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:22.378120    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:27.382275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:27.382241    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:32:27.382890 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:27.382815    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:37.383593 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:37.383548    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:47.383432 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:47.383375    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:55.113363 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:55.113332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:57.383145 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:57.383107    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:59.772625 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:59.772592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:59.872710 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:32:59.872678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.488501 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.488464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.589431 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.589395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.607762 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.607735    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:33:04.607970 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.607947    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container" containerID="cri-o://7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b" gracePeriod=30
Sep 17 23:33:04.608028 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.607984    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kube-rbac-proxy" containerID="cri-o://c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd" gracePeriod=30
Sep 17 23:33:04.696355 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.696313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.795453 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.795390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.818689 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.818644    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:33:04.819055 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819037    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container"
Sep 17 23:33:04.819132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819057    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container"
Sep 17 23:33:04.819132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819093    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kube-rbac-proxy"
Sep 17 23:33:04.819132 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819102    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kube-rbac-proxy"
Sep 17 23:33:04.819287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819187    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kserve-container"
Sep 17 23:33:04.819287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.819199    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb23a376-b313-42e2-82cb-2ae293396b0f" containerName="kube-rbac-proxy"
Sep 17 23:33:04.821764 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.821745    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:04.826129 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.826109    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a1f9-predictor-serving-cert\""
Sep 17 23:33:04.826234 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.826112    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a1f9-kube-rbac-proxy-sar-config\""
Sep 17 23:33:04.835185 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.835162    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:33:04.869045 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.869015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:04.923183 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.923162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:04.923275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.923208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:04.923275 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.923239    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kls\" (UniqueName: \"kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:04.987554 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:04.987533    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:05.023595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.023572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kls\" (UniqueName: \"kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.023706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.023622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.023706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.023655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.024329 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.024310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.026060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.026032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.034089 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.034071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kls\" (UniqueName: \"kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls\") pod \"error-404-isvc-4a1f9-predictor-555b74f697-hp8h5\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") " pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.131737 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.131680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.418024 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.417964    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:33:05.421334 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:33:05.421308    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78fdfc0_af40_40ba_8682_fb61e1ef4866.slice/crio-3ba9dd2c24f9ca0351e9c50cee397d11d0cb6e5a26c03043e0769675adce4b28 WatchSource:0}: Error finding container 3ba9dd2c24f9ca0351e9c50cee397d11d0cb6e5a26c03043e0769675adce4b28: Status 404 returned error can't find the container with id 3ba9dd2c24f9ca0351e9c50cee397d11d0cb6e5a26c03043e0769675adce4b28
Sep 17 23:33:05.576064 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.576031    2559 generic.go:356] "Generic (PLEG): container finished" podID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerID="c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd" exitCode=2
Sep 17 23:33:05.576352 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.576096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerDied","Data":"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"}
Sep 17 23:33:05.577833 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.577814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerStarted","Data":"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"}
Sep 17 23:33:05.577920 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.577853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerStarted","Data":"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"}
Sep 17 23:33:05.577920 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.577874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerStarted","Data":"3ba9dd2c24f9ca0351e9c50cee397d11d0cb6e5a26c03043e0769675adce4b28"}
Sep 17 23:33:05.578012 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.577958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:05.606776 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:05.606721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podStartSLOduration=1.6067008280000001 podStartE2EDuration="1.606700828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:05.603554742 +0000 UTC m=+1340.411665892" watchObservedRunningTime="2026-09-17 23:33:05.606700828 +0000 UTC m=+1340.414811979"
Sep 17 23:33:06.583373 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:06.583328    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:06.584545 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:06.584505    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:07.384192 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.384166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:33:07.467021 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.467003    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:33:07.543162 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.543108    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config\") pod \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") "
Sep 17 23:33:07.543294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.543169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls\") pod \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") "
Sep 17 23:33:07.543294 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.543216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-967sb\" (UniqueName: \"kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb\") pod \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\" (UID: \"ec6ad9c8-ca76-4742-b02c-44393a72e6b6\") "
Sep 17 23:33:07.543488 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.543462    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec6ad9c8-ca76-4742-b02c-44393a72e6b6" (UID: "ec6ad9c8-ca76-4742-b02c-44393a72e6b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:07.545229 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.545204    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb" (OuterVolumeSpecName: "kube-api-access-967sb") pod "ec6ad9c8-ca76-4742-b02c-44393a72e6b6" (UID: "ec6ad9c8-ca76-4742-b02c-44393a72e6b6"). InnerVolumeSpecName "kube-api-access-967sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:07.545329 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.545287    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec6ad9c8-ca76-4742-b02c-44393a72e6b6" (UID: "ec6ad9c8-ca76-4742-b02c-44393a72e6b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:07.590741 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.590711    2559 generic.go:356] "Generic (PLEG): container finished" podID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerID="7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b" exitCode=0
Sep 17 23:33:07.591051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.590759    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"
Sep 17 23:33:07.591051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.590796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerDied","Data":"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"}
Sep 17 23:33:07.591051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.590846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq" event={"ID":"ec6ad9c8-ca76-4742-b02c-44393a72e6b6","Type":"ContainerDied","Data":"de5e8a3f808468fdbf4543c6fbcb14c19538b4a08e3fbde21039dee3a12e93bb"}
Sep 17 23:33:07.591051 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.590871    2559 scope.go:117] "RemoveContainer" containerID="c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"
Sep 17 23:33:07.591471 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.591444    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:07.599456 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.599440    2559 scope.go:117] "RemoveContainer" containerID="7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"
Sep 17 23:33:07.606522 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.606505    2559 scope.go:117] "RemoveContainer" containerID="c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"
Sep 17 23:33:07.606813 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:33:07.606794    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd\": container with ID starting with c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd not found: ID does not exist" containerID="c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"
Sep 17 23:33:07.606887 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.606826    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd"} err="failed to get container status \"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd\": rpc error: code = NotFound desc = could not find container \"c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd\": container with ID starting with c9db49c9ee4803c75200abfd007f6caffd1f693639c082550905f3874c0bc6cd not found: ID does not exist"
Sep 17 23:33:07.606887 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.606849    2559 scope.go:117] "RemoveContainer" containerID="7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"
Sep 17 23:33:07.607097 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:33:07.607079    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b\": container with ID starting with 7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b not found: ID does not exist" containerID="7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"
Sep 17 23:33:07.607139 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.607103    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b"} err="failed to get container status \"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b\": rpc error: code = NotFound desc = could not find container \"7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b\": container with ID starting with 7dc898ed86b4c03a1d80b9651bca262f8c8f72d666703aa5c341adf7b6e0d01b not found: ID does not exist"
Sep 17 23:33:07.614691 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.614671    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:33:07.620057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.620037    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2026c-predictor-5f7545dbd9-9k6vq"]
Sep 17 23:33:07.643821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.643801    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:07.643821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.643821    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:07.643980 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.643833    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967sb\" (UniqueName: \"kubernetes.io/projected/ec6ad9c8-ca76-4742-b02c-44393a72e6b6-kube-api-access-967sb\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:07.648751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:07.648732    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" path="/var/lib/kubelet/pods/ec6ad9c8-ca76-4742-b02c-44393a72e6b6/volumes"
Sep 17 23:33:12.595454 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:12.595373    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:12.595918 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:12.595888    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:18.261595 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:18.261554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:18.410161 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:18.410119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:22.595900 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:22.595856    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:28.347802 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.347687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.478297 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.478267    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:33:28.478563 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.478535    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container" containerID="cri-o://c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5" gracePeriod=30
Sep 17 23:33:28.478645 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.478546    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kube-rbac-proxy" containerID="cri-o://f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc" gracePeriod=30
Sep 17 23:33:28.492147 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.492122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.566706 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.566676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.653883 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.653815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.668262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668236    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:33:28.668558 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668547    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container"
Sep 17 23:33:28.668601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668560    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container"
Sep 17 23:33:28.668601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668575    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kube-rbac-proxy"
Sep 17 23:33:28.668601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668580    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kube-rbac-proxy"
Sep 17 23:33:28.668728 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668643    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kserve-container"
Sep 17 23:33:28.668728 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.668651    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6ad9c8-ca76-4742-b02c-44393a72e6b6" containerName="kube-rbac-proxy"
Sep 17 23:33:28.671702 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.671682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.674404 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.674387    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6263-predictor-serving-cert\""
Sep 17 23:33:28.674517 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.674425    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d6263-kube-rbac-proxy-sar-config\""
Sep 17 23:33:28.686484 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.686413    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:33:28.687738 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.687712    2559 generic.go:356] "Generic (PLEG): container finished" podID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerID="f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc" exitCode=2
Sep 17 23:33:28.687850 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.687749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerDied","Data":"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"}
Sep 17 23:33:28.769510 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.769469    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.793524 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.793500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7q7\" (UniqueName: \"kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.793616 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.793568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.793680 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.793646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.894948 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.894922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7q7\" (UniqueName: \"kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.895042 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.895003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.895086 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.895047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.895827 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.895808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.897328 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.897307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.903868 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.903850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7q7\" (UniqueName: \"kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7\") pod \"error-404-isvc-d6263-predictor-5f7487c565-5vmd2\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") " pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:28.984912 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:28.984888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:29.262062 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.261995    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:33:29.265304 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:33:29.265280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7656f818_9d96_4f57_8334_ff021de9d720.slice/crio-7914ffa53bcd283d784622c28e7c83896e6d5d97a3a9ad524f3d6d6f7063cda7 WatchSource:0}: Error finding container 7914ffa53bcd283d784622c28e7c83896e6d5d97a3a9ad524f3d6d6f7063cda7: Status 404 returned error can't find the container with id 7914ffa53bcd283d784622c28e7c83896e6d5d97a3a9ad524f3d6d6f7063cda7
Sep 17 23:33:29.602592 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.602518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:29.694088 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.694061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerStarted","Data":"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"}
Sep 17 23:33:29.694240 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.694115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:29.694240 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.694137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerStarted","Data":"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"}
Sep 17 23:33:29.694240 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.694159    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:29.694240 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.694185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerStarted","Data":"7914ffa53bcd283d784622c28e7c83896e6d5d97a3a9ad524f3d6d6f7063cda7"}
Sep 17 23:33:29.695613 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.695584    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:33:29.712738 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:29.712693    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podStartSLOduration=1.712677823 podStartE2EDuration="1.712677823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:29.711087766 +0000 UTC m=+1364.519198917" watchObservedRunningTime="2026-09-17 23:33:29.712677823 +0000 UTC m=+1364.520788963"
Sep 17 23:33:30.699078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:30.699043    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:33:31.434219 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.434166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:33:31.523465 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.523426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config\") pod \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") "
Sep 17 23:33:31.523597 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.523479    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5676\" (UniqueName: \"kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676\") pod \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") "
Sep 17 23:33:31.523597 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.523531    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") pod \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\" (UID: \"4a19abef-acb7-4683-9c35-7563bb6cb8f4\") "
Sep 17 23:33:31.523779 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.523757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a19abef-acb7-4683-9c35-7563bb6cb8f4" (UID: "4a19abef-acb7-4683-9c35-7563bb6cb8f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:31.525412 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.525386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676" (OuterVolumeSpecName: "kube-api-access-p5676") pod "4a19abef-acb7-4683-9c35-7563bb6cb8f4" (UID: "4a19abef-acb7-4683-9c35-7563bb6cb8f4"). InnerVolumeSpecName "kube-api-access-p5676". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:31.525518 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.525459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a19abef-acb7-4683-9c35-7563bb6cb8f4" (UID: "4a19abef-acb7-4683-9c35-7563bb6cb8f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:31.623944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.623922    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a19abef-acb7-4683-9c35-7563bb6cb8f4-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.623944 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.623944    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.624068 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.623955    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5676\" (UniqueName: \"kubernetes.io/projected/4a19abef-acb7-4683-9c35-7563bb6cb8f4-kube-api-access-p5676\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.704016 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.703960    2559 generic.go:356] "Generic (PLEG): container finished" podID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerID="c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5" exitCode=0
Sep 17 23:33:31.704016 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.704010    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"
Sep 17 23:33:31.704403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.704033    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerDied","Data":"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"}
Sep 17 23:33:31.704403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.704097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n" event={"ID":"4a19abef-acb7-4683-9c35-7563bb6cb8f4","Type":"ContainerDied","Data":"4ece32449b1e560bb8c3dca1adfb0ff8bc5bf3072a9e1759e218ac2ec875d8dd"}
Sep 17 23:33:31.704403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.704138    2559 scope.go:117] "RemoveContainer" containerID="f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"
Sep 17 23:33:31.712296 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.712279    2559 scope.go:117] "RemoveContainer" containerID="c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"
Sep 17 23:33:31.719580 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.719560    2559 scope.go:117] "RemoveContainer" containerID="f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"
Sep 17 23:33:31.720338 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:33:31.720217    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc\": container with ID starting with f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc not found: ID does not exist" containerID="f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"
Sep 17 23:33:31.720338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.720250    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc"} err="failed to get container status \"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc\": rpc error: code = NotFound desc = could not find container \"f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc\": container with ID starting with f339df3b1f755b835f01ba1351f055f3111e702034bc24bc0a3ad17d9c797bbc not found: ID does not exist"
Sep 17 23:33:31.720338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.720272    2559 scope.go:117] "RemoveContainer" containerID="c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"
Sep 17 23:33:31.721076 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:33:31.721036    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5\": container with ID starting with c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5 not found: ID does not exist" containerID="c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"
Sep 17 23:33:31.721253 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.721083    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5"} err="failed to get container status \"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5\": rpc error: code = NotFound desc = could not find container \"c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5\": container with ID starting with c910a713b8e6cbe0474ecb4c4ed37ea50e4d0b9d40fcce3b77fa87c3211e97d5 not found: ID does not exist"
Sep 17 23:33:31.721253 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.721191    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:33:31.723220 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:31.723200    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3e634-predictor-78db59dc7-blq7n"]
Sep 17 23:33:32.596811 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:32.596775    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:33.648484 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:33.648452    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" path="/var/lib/kubelet/pods/4a19abef-acb7-4683-9c35-7563bb6cb8f4/volumes"
Sep 17 23:33:35.703808 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:35.703779    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:33:35.704232 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:35.704163    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:33:42.596160 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:42.596122    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:45.704965 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:45.704924    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:33:52.596528 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:52.596496    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:33:55.704290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:33:55.704243    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:34:04.683922 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:04.683885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:05.562835 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:05.562798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:05.704976 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:05.704937    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:34:14.786710 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.786654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:14.900718 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.900684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:14.908667 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.908615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:34:14.908949 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.908918    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container" containerID="cri-o://697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899" gracePeriod=30
Sep 17 23:34:14.909040 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.908981    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kube-rbac-proxy" containerID="cri-o://78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340" gracePeriod=30
Sep 17 23:34:14.994019 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:14.993986    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:15.073392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.073305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:15.099021 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.098985    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:34:15.099328 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099316    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kube-rbac-proxy"
Sep 17 23:34:15.099379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099330    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kube-rbac-proxy"
Sep 17 23:34:15.099379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container"
Sep 17 23:34:15.099379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099345    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container"
Sep 17 23:34:15.099497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099407    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kube-rbac-proxy"
Sep 17 23:34:15.099497 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.099416    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a19abef-acb7-4683-9c35-7563bb6cb8f4" containerName="kserve-container"
Sep 17 23:34:15.102498 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.102480    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.104816 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.104797    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-51963-predictor-serving-cert\""
Sep 17 23:34:15.104920 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.104876    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-51963-kube-rbac-proxy-sar-config\""
Sep 17 23:34:15.115655 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.115630    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:34:15.149390 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.149356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:15.263254 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.263215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.263426 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.263283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.263426 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.263404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d726t\" (UniqueName: \"kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.306333 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.306303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:15.364864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.364794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d726t\" (UniqueName: \"kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.364864 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.364839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.365021 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.364894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.365072 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:34:15.365022    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-51963-predictor-serving-cert: secret "error-404-isvc-51963-predictor-serving-cert" not found
Sep 17 23:34:15.365109 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:34:15.365097    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls podName:f217de27-ace7-42db-95e9-c8889c017890 nodeName:}" failed. No retries permitted until 2026-09-17 23:34:15.865080547 +0000 UTC m=+1410.673191681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls") pod "error-404-isvc-51963-predictor-5b7dc99599-gr8vf" (UID: "f217de27-ace7-42db-95e9-c8889c017890") : secret "error-404-isvc-51963-predictor-serving-cert" not found
Sep 17 23:34:15.365573 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.365556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.376737 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.376710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d726t\" (UniqueName: \"kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.436984 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.436949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:15.705403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.705376    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:34:15.869037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.869000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.874712 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.872308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") pod \"error-404-isvc-51963-predictor-5b7dc99599-gr8vf\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") " pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:15.903619 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.903585    2559 generic.go:356] "Generic (PLEG): container finished" podID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerID="78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340" exitCode=2
Sep 17 23:34:15.903788 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:15.903678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerDied","Data":"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"}
Sep 17 23:34:16.014932 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.014832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:16.288229 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.288200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:34:16.290741 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:34:16.290713    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf217de27_ace7_42db_95e9_c8889c017890.slice/crio-758ad466285a06d34eccbeaefbea15c5fd96d8c3dff58ec4cde833b4bb0d33a6 WatchSource:0}: Error finding container 758ad466285a06d34eccbeaefbea15c5fd96d8c3dff58ec4cde833b4bb0d33a6: Status 404 returned error can't find the container with id 758ad466285a06d34eccbeaefbea15c5fd96d8c3dff58ec4cde833b4bb0d33a6
Sep 17 23:34:16.909492 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.909454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerStarted","Data":"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"}
Sep 17 23:34:16.909903 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.909524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerStarted","Data":"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"}
Sep 17 23:34:16.909903 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.909552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerStarted","Data":"758ad466285a06d34eccbeaefbea15c5fd96d8c3dff58ec4cde833b4bb0d33a6"}
Sep 17 23:34:16.909903 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.909583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:16.929252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:16.929209    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podStartSLOduration=1.929195983 podStartE2EDuration="1.929195983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:16.92775211 +0000 UTC m=+1411.735863261" watchObservedRunningTime="2026-09-17 23:34:16.929195983 +0000 UTC m=+1411.737307184"
Sep 17 23:34:17.591403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:17.591356    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 23:34:17.914768 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:17.914675    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:17.916124 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:17.916093    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:34:18.253559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.253535    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:34:18.391601 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.391565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config\") pod \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") "
Sep 17 23:34:18.391796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.391612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls\") pod \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") "
Sep 17 23:34:18.391796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.391680    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8kls\" (UniqueName: \"kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls\") pod \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\" (UID: \"e78fdfc0-af40-40ba-8682-fb61e1ef4866\") "
Sep 17 23:34:18.391997 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.391966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e78fdfc0-af40-40ba-8682-fb61e1ef4866" (UID: "e78fdfc0-af40-40ba-8682-fb61e1ef4866"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:18.393742 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.393718    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e78fdfc0-af40-40ba-8682-fb61e1ef4866" (UID: "e78fdfc0-af40-40ba-8682-fb61e1ef4866"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:18.393876 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.393863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls" (OuterVolumeSpecName: "kube-api-access-d8kls") pod "e78fdfc0-af40-40ba-8682-fb61e1ef4866" (UID: "e78fdfc0-af40-40ba-8682-fb61e1ef4866"). InnerVolumeSpecName "kube-api-access-d8kls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:18.492194 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.492165    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kls\" (UniqueName: \"kubernetes.io/projected/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-api-access-d8kls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:34:18.492194 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.492191    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e78fdfc0-af40-40ba-8682-fb61e1ef4866-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:34:18.492380 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.492201    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e78fdfc0-af40-40ba-8682-fb61e1ef4866-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:34:18.920264 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920168    2559 generic.go:356] "Generic (PLEG): container finished" podID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerID="697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899" exitCode=0
Sep 17 23:34:18.920264 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920237    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"
Sep 17 23:34:18.920801 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerDied","Data":"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"}
Sep 17 23:34:18.920801 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5" event={"ID":"e78fdfc0-af40-40ba-8682-fb61e1ef4866","Type":"ContainerDied","Data":"3ba9dd2c24f9ca0351e9c50cee397d11d0cb6e5a26c03043e0769675adce4b28"}
Sep 17 23:34:18.920801 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920356    2559 scope.go:117] "RemoveContainer" containerID="78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"
Sep 17 23:34:18.920972 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.920911    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:34:18.928762 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.928741    2559 scope.go:117] "RemoveContainer" containerID="697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"
Sep 17 23:34:18.935992 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.935975    2559 scope.go:117] "RemoveContainer" containerID="78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"
Sep 17 23:34:18.936252 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:34:18.936235    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340\": container with ID starting with 78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340 not found: ID does not exist" containerID="78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"
Sep 17 23:34:18.936298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.936261    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340"} err="failed to get container status \"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340\": rpc error: code = NotFound desc = could not find container \"78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340\": container with ID starting with 78b4f4b6ad0a7b95a73abaa93f99c88fc7a41ef3a578b8e01450c204896e9340 not found: ID does not exist"
Sep 17 23:34:18.936298 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.936279    2559 scope.go:117] "RemoveContainer" containerID="697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"
Sep 17 23:34:18.936528 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:34:18.936511    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899\": container with ID starting with 697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899 not found: ID does not exist" containerID="697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"
Sep 17 23:34:18.936572 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.936535    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899"} err="failed to get container status \"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899\": rpc error: code = NotFound desc = could not find container \"697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899\": container with ID starting with 697c35c2f09177ebb131047798c7d6f716caea677f39d72e61308511249ba899 not found: ID does not exist"
Sep 17 23:34:18.941264 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.941242    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:34:18.947087 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:18.947067    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a1f9-predictor-555b74f697-hp8h5"]
Sep 17 23:34:19.648914 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:19.648881    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" path="/var/lib/kubelet/pods/e78fdfc0-af40-40ba-8682-fb61e1ef4866/volumes"
Sep 17 23:34:23.924814 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:23.924783    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:34:23.925391 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:23.925365    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:34:28.600442 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:28.600405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:28.760519 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:28.760483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:33.925886 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:33.925846    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:34:43.925357 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:43.925269    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:34:48.174741 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:48.174653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:53.925873 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:34:53.925830    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 23:35:03.926403 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:35:03.926364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:35:15.034523 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:35:15.034484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:15.489454 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:35:15.489416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:29.385280 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:35:29.385236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:40.733160 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:35:40.733115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:45.206650 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:39:45.206612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:52.598134 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:41:52.598093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.378721 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.378678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.534287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.534255    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.622522 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.622486    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:42:43.622766 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.622736    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" containerID="cri-o://6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74" gracePeriod=30
Sep 17 23:42:43.622916 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.622788    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kube-rbac-proxy" containerID="cri-o://bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1" gracePeriod=30
Sep 17 23:42:43.641465 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.641439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.677373 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677347    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:42:43.677842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677824    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kube-rbac-proxy"
Sep 17 23:42:43.677842 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677843    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kube-rbac-proxy"
Sep 17 23:42:43.678001 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677857    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container"
Sep 17 23:42:43.678001 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677871    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container"
Sep 17 23:42:43.678001 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.677989    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kserve-container"
Sep 17 23:42:43.678172 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.678005    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78fdfc0-af40-40ba-8682-fb61e1ef4866" containerName="kube-rbac-proxy"
Sep 17 23:42:43.681213 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.681196    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.684801 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.684783    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc189-predictor-serving-cert\""
Sep 17 23:42:43.685423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.685399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fc189-kube-rbac-proxy-sar-config\""
Sep 17 23:42:43.695797 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.695774    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:42:43.739400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.739382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.785676 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.785640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.785763 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.785701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.785813 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.785797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7svx\" (UniqueName: \"kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.839078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.839054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:43.886422 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.886369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7svx\" (UniqueName: \"kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.886489 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.886467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.886540 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.886498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.887057 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.887040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.888810 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.888791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.894963 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.894947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7svx\" (UniqueName: \"kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx\") pod \"error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") " pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:43.991374 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:43.991354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:44.224777 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:44.224745    2559 generic.go:356] "Generic (PLEG): container finished" podID="7656f818-9d96-4f57-8334-ff021de9d720" containerID="bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1" exitCode=2
Sep 17 23:42:44.224910 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:44.224815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerDied","Data":"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"}
Sep 17 23:42:44.264351 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:44.264325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:42:44.268382 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:42:44.268359    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9b3305_224f_4a35_b878_14d7b23d2c19.slice/crio-137d7f0e3f88ef2ab0cedbbd39a0cd9b0f18feec09f636cf283183ea6f7b365a WatchSource:0}: Error finding container 137d7f0e3f88ef2ab0cedbbd39a0cd9b0f18feec09f636cf283183ea6f7b365a: Status 404 returned error can't find the container with id 137d7f0e3f88ef2ab0cedbbd39a0cd9b0f18feec09f636cf283183ea6f7b365a
Sep 17 23:42:45.230300 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.230263    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerStarted","Data":"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"}
Sep 17 23:42:45.230686 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.230328    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:45.230686 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.230353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerStarted","Data":"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"}
Sep 17 23:42:45.230686 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.230377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerStarted","Data":"137d7f0e3f88ef2ab0cedbbd39a0cd9b0f18feec09f636cf283183ea6f7b365a"}
Sep 17 23:42:45.230686 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.230397    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:45.231815 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.231788    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:42:45.252744 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.252689    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podStartSLOduration=2.252646808 podStartE2EDuration="2.252646808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:45.252118968 +0000 UTC m=+1920.060230119" watchObservedRunningTime="2026-09-17 23:42:45.252646808 +0000 UTC m=+1920.060757959"
Sep 17 23:42:45.699753 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.699720    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 23:42:45.705372 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:45.705092    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 23:42:46.235751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.235711    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:42:46.469338 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.469316    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:42:46.507423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.507363    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl7q7\" (UniqueName: \"kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7\") pod \"7656f818-9d96-4f57-8334-ff021de9d720\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") "
Sep 17 23:42:46.507423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.507393    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls\") pod \"7656f818-9d96-4f57-8334-ff021de9d720\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") "
Sep 17 23:42:46.507423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.507415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config\") pod \"7656f818-9d96-4f57-8334-ff021de9d720\" (UID: \"7656f818-9d96-4f57-8334-ff021de9d720\") "
Sep 17 23:42:46.507796 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.507773    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7656f818-9d96-4f57-8334-ff021de9d720" (UID: "7656f818-9d96-4f57-8334-ff021de9d720"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:46.509256 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.509233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7" (OuterVolumeSpecName: "kube-api-access-dl7q7") pod "7656f818-9d96-4f57-8334-ff021de9d720" (UID: "7656f818-9d96-4f57-8334-ff021de9d720"). InnerVolumeSpecName "kube-api-access-dl7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:46.509379 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.509315    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7656f818-9d96-4f57-8334-ff021de9d720" (UID: "7656f818-9d96-4f57-8334-ff021de9d720"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:46.608866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.608845    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7q7\" (UniqueName: \"kubernetes.io/projected/7656f818-9d96-4f57-8334-ff021de9d720-kube-api-access-dl7q7\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:42:46.608866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.608864    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7656f818-9d96-4f57-8334-ff021de9d720-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:42:46.608994 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:46.608876    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7656f818-9d96-4f57-8334-ff021de9d720-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:42:47.241573 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.241541    2559 generic.go:356] "Generic (PLEG): container finished" podID="7656f818-9d96-4f57-8334-ff021de9d720" containerID="6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74" exitCode=0
Sep 17 23:42:47.242054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.241597    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"
Sep 17 23:42:47.242054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.241608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerDied","Data":"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"}
Sep 17 23:42:47.242054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.241652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2" event={"ID":"7656f818-9d96-4f57-8334-ff021de9d720","Type":"ContainerDied","Data":"7914ffa53bcd283d784622c28e7c83896e6d5d97a3a9ad524f3d6d6f7063cda7"}
Sep 17 23:42:47.242054 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.241705    2559 scope.go:117] "RemoveContainer" containerID="bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"
Sep 17 23:42:47.249894 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.249875    2559 scope.go:117] "RemoveContainer" containerID="6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"
Sep 17 23:42:47.256683 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.256650    2559 scope.go:117] "RemoveContainer" containerID="bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"
Sep 17 23:42:47.256940 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:42:47.256922    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1\": container with ID starting with bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1 not found: ID does not exist" containerID="bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"
Sep 17 23:42:47.256999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.256948    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1"} err="failed to get container status \"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1\": rpc error: code = NotFound desc = could not find container \"bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1\": container with ID starting with bf3ea1e52f8a5a1e0669b68c6a09f5fd09093f6ba81f133a13db53097e594dc1 not found: ID does not exist"
Sep 17 23:42:47.256999 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.256964    2559 scope.go:117] "RemoveContainer" containerID="6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"
Sep 17 23:42:47.257208 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:42:47.257192    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74\": container with ID starting with 6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74 not found: ID does not exist" containerID="6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"
Sep 17 23:42:47.257247 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.257212    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74"} err="failed to get container status \"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74\": rpc error: code = NotFound desc = could not find container \"6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74\": container with ID starting with 6b409de769b6d57f602e8d48caf394bc7c76bac41e262b97bc85d2d3fa48fd74 not found: ID does not exist"
Sep 17 23:42:47.277751 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.277723    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:42:47.299849 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.299828    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d6263-predictor-5f7487c565-5vmd2"]
Sep 17 23:42:47.648622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:47.648553    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7656f818-9d96-4f57-8334-ff021de9d720" path="/var/lib/kubelet/pods/7656f818-9d96-4f57-8334-ff021de9d720/volumes"
Sep 17 23:42:51.239919 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:51.239894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:42:51.240359 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:42:51.240326    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:43:01.240531 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:01.240490    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:43:11.241096 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:11.241052    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:43:21.241078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:21.241032    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 23:43:29.721310 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:29.721274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:29.902880 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:29.902848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:29.911104 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:29.911073    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:43:29.911330 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:29.911307    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container" containerID="cri-o://d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24" gracePeriod=30
Sep 17 23:43:29.911411 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:29.911326    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kube-rbac-proxy" containerID="cri-o://73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347" gracePeriod=30
Sep 17 23:43:30.000156 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.000078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:30.096715 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.096656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:30.101037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101012    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 17 23:43:30.101337 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101325    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kube-rbac-proxy"
Sep 17 23:43:30.101402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101339    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kube-rbac-proxy"
Sep 17 23:43:30.101402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101352    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container"
Sep 17 23:43:30.101402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101358    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container"
Sep 17 23:43:30.101559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101424    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kserve-container"
Sep 17 23:43:30.101559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.101431    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7656f818-9d96-4f57-8334-ff021de9d720" containerName="kube-rbac-proxy"
Sep 17 23:43:30.104414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.104398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.106773 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.106748    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-57566-kube-rbac-proxy-sar-config\""
Sep 17 23:43:30.106886 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.106768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-57566-predictor-serving-cert\""
Sep 17 23:43:30.115044 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.115021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 17 23:43:30.164299 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.164273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:30.253843 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.253790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:30.285287 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.285266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.285414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.285302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.285414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.285322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzp8r\" (UniqueName: \"kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.386560 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.386530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.386695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.386628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.386695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.386688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzp8r\" (UniqueName: \"kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.387349 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.387326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.388970 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.388950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.403614 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.403592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzp8r\" (UniqueName: \"kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r\") pod \"error-404-isvc-57566-predictor-85dbbdfff7-k9s8n\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") " pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.416402 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.416379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:30.442641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.442616    2559 generic.go:356] "Generic (PLEG): container finished" podID="f217de27-ace7-42db-95e9-c8889c017890" containerID="73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347" exitCode=2
Sep 17 23:43:30.442767 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.442695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerDied","Data":"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"}
Sep 17 23:43:30.451365 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.451338    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:30.691819 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:30.691791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 17 23:43:30.694397 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:43:30.694368    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77b7875_e9e9_4a15_8f78_16f94410a879.slice/crio-e5b6a1920f33d457596e45d4bc051aae6de1190ecde6125d45950905cf443d13 WatchSource:0}: Error finding container e5b6a1920f33d457596e45d4bc051aae6de1190ecde6125d45950905cf443d13: Status 404 returned error can't find the container with id e5b6a1920f33d457596e45d4bc051aae6de1190ecde6125d45950905cf443d13
Sep 17 23:43:31.240922 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.240894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:43:31.450045 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.450010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerStarted","Data":"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"}
Sep 17 23:43:31.450207 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.450065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerStarted","Data":"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"}
Sep 17 23:43:31.450207 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.450084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerStarted","Data":"e5b6a1920f33d457596e45d4bc051aae6de1190ecde6125d45950905cf443d13"}
Sep 17 23:43:31.450207 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.450178    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:31.470797 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:31.470754    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podStartSLOduration=1.4707401039999999 podStartE2EDuration="1.470740104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:31.468769223 +0000 UTC m=+1966.276880372" watchObservedRunningTime="2026-09-17 23:43:31.470740104 +0000 UTC m=+1966.278851253"
Sep 17 23:43:32.454583 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.454552    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:32.455806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.455777    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:43:32.755079 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.755059    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:43:32.806795 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.806770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config\") pod \"f217de27-ace7-42db-95e9-c8889c017890\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") "
Sep 17 23:43:32.806935 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.806800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") pod \"f217de27-ace7-42db-95e9-c8889c017890\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") "
Sep 17 23:43:32.806935 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.806828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d726t\" (UniqueName: \"kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t\") pod \"f217de27-ace7-42db-95e9-c8889c017890\" (UID: \"f217de27-ace7-42db-95e9-c8889c017890\") "
Sep 17 23:43:32.807162 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.807137    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f217de27-ace7-42db-95e9-c8889c017890" (UID: "f217de27-ace7-42db-95e9-c8889c017890"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:32.808650 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.808622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f217de27-ace7-42db-95e9-c8889c017890" (UID: "f217de27-ace7-42db-95e9-c8889c017890"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:43:32.808857 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.808833    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t" (OuterVolumeSpecName: "kube-api-access-d726t") pod "f217de27-ace7-42db-95e9-c8889c017890" (UID: "f217de27-ace7-42db-95e9-c8889c017890"). InnerVolumeSpecName "kube-api-access-d726t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:43:32.908085 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.908062    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f217de27-ace7-42db-95e9-c8889c017890-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:43:32.908085 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.908083    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f217de27-ace7-42db-95e9-c8889c017890-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:43:32.908208 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:32.908094    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d726t\" (UniqueName: \"kubernetes.io/projected/f217de27-ace7-42db-95e9-c8889c017890-kube-api-access-d726t\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:43:33.463197 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463166    2559 generic.go:356] "Generic (PLEG): container finished" podID="f217de27-ace7-42db-95e9-c8889c017890" containerID="d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24" exitCode=0
Sep 17 23:43:33.463641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463223    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"
Sep 17 23:43:33.463641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerDied","Data":"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"}
Sep 17 23:43:33.463641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf" event={"ID":"f217de27-ace7-42db-95e9-c8889c017890","Type":"ContainerDied","Data":"758ad466285a06d34eccbeaefbea15c5fd96d8c3dff58ec4cde833b4bb0d33a6"}
Sep 17 23:43:33.463641 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463336    2559 scope.go:117] "RemoveContainer" containerID="73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"
Sep 17 23:43:33.463913 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.463879    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:43:33.472120 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.472093    2559 scope.go:117] "RemoveContainer" containerID="d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"
Sep 17 23:43:33.479131 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.479114    2559 scope.go:117] "RemoveContainer" containerID="73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"
Sep 17 23:43:33.479387 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:43:33.479368    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347\": container with ID starting with 73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347 not found: ID does not exist" containerID="73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"
Sep 17 23:43:33.479453 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.479394    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347"} err="failed to get container status \"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347\": rpc error: code = NotFound desc = could not find container \"73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347\": container with ID starting with 73091f53cf3518fb19150db96080b3a127c4c556f0636d5c5e6752e0b5993347 not found: ID does not exist"
Sep 17 23:43:33.479453 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.479409    2559 scope.go:117] "RemoveContainer" containerID="d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"
Sep 17 23:43:33.479631 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:43:33.479612    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24\": container with ID starting with d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24 not found: ID does not exist" containerID="d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"
Sep 17 23:43:33.479697 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.479639    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24"} err="failed to get container status \"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24\": rpc error: code = NotFound desc = could not find container \"d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24\": container with ID starting with d46286edab1c2a1b6a609b90cdd6cb64621452eeb2a30864de18517c773fbc24 not found: ID does not exist"
Sep 17 23:43:33.486793 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.486768    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:43:33.488823 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.488804    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-51963-predictor-5b7dc99599-gr8vf"]
Sep 17 23:43:33.648695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:33.648649    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f217de27-ace7-42db-95e9-c8889c017890" path="/var/lib/kubelet/pods/f217de27-ace7-42db-95e9-c8889c017890/volumes"
Sep 17 23:43:38.468581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:38.468510    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:43:38.468981 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:38.468956    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:43:48.469570 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:48.469531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:43:53.610689 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:53.610582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:53.891424 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:53.891339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:58.469019 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:43:58.468980    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:44:03.681036 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.680992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:03.792830 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.792787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:03.798423 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.798390    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:44:03.798642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.798616    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container" containerID="cri-o://bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a" gracePeriod=30
Sep 17 23:44:03.798642 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.798630    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kube-rbac-proxy" containerID="cri-o://09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0" gracePeriod=30
Sep 17 23:44:03.870420 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:03.870380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.029716 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.029587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.060575 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060541    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:44:04.060920 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060906    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container"
Sep 17 23:44:04.060966 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060922    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container"
Sep 17 23:44:04.060966 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060938    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kube-rbac-proxy"
Sep 17 23:44:04.060966 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060944    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kube-rbac-proxy"
Sep 17 23:44:04.061061 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.060991    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kserve-container"
Sep 17 23:44:04.061061 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.061001    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f217de27-ace7-42db-95e9-c8889c017890" containerName="kube-rbac-proxy"
Sep 17 23:44:04.064100 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.064079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.066913 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.066890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daa97-predictor-serving-cert\""
Sep 17 23:44:04.067609 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.067591    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daa97-kube-rbac-proxy-sar-config\""
Sep 17 23:44:04.078096 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.078062    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:44:04.133518 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.133479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.163884 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.163845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwm7h\" (UniqueName: \"kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.163884 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.163887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.164088 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.163957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.230030 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.229992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.265220 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.265177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwm7h\" (UniqueName: \"kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.265400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.265228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.265400 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.265307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.266624 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.266593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.272683 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.269225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.275548 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.275525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwm7h\" (UniqueName: \"kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h\") pod \"error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") " pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.375065 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.374964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:04.535019 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.534978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.601516 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.601483    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerID="09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0" exitCode=2
Sep 17 23:44:04.601687 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.601556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerDied","Data":"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"}
Sep 17 23:44:04.653251 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:04.653223    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:44:04.655594 ip-10-0-128-70 kubenswrapper[2559]: W0917 23:44:04.655567    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62ed706_b889_47f5_86dc_2cf982f74298.slice/crio-6c71d35467cf5d737139bec4528ebb35f49c04a4f3eddc8ef74439cd1d3dad27 WatchSource:0}: Error finding container 6c71d35467cf5d737139bec4528ebb35f49c04a4f3eddc8ef74439cd1d3dad27: Status 404 returned error can't find the container with id 6c71d35467cf5d737139bec4528ebb35f49c04a4f3eddc8ef74439cd1d3dad27
Sep 17 23:44:05.607806 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.607773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerStarted","Data":"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"}
Sep 17 23:44:05.608252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.607842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerStarted","Data":"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"}
Sep 17 23:44:05.608252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.607872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerStarted","Data":"6c71d35467cf5d737139bec4528ebb35f49c04a4f3eddc8ef74439cd1d3dad27"}
Sep 17 23:44:05.608252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.607914    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:05.608252 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.607944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:05.609867 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.609833    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:05.628174 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:05.628117    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podStartSLOduration=1.62810153 podStartE2EDuration="1.62810153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:05.62685124 +0000 UTC m=+2000.434962392" watchObservedRunningTime="2026-09-17 23:44:05.62810153 +0000 UTC m=+2000.436212680"
Sep 17 23:44:06.236613 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:06.236562    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 23:44:06.612852 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:06.612762    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:07.243462 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.243439    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:44:07.394557 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.394469    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config\") pod \"7c9b3305-224f-4a35-b878-14d7b23d2c19\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") "
Sep 17 23:44:07.394557 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.394519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls\") pod \"7c9b3305-224f-4a35-b878-14d7b23d2c19\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") "
Sep 17 23:44:07.394866 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.394568    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7svx\" (UniqueName: \"kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx\") pod \"7c9b3305-224f-4a35-b878-14d7b23d2c19\" (UID: \"7c9b3305-224f-4a35-b878-14d7b23d2c19\") "
Sep 17 23:44:07.394937 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.394864    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c9b3305-224f-4a35-b878-14d7b23d2c19" (UID: "7c9b3305-224f-4a35-b878-14d7b23d2c19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:07.396602 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.396582    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx" (OuterVolumeSpecName: "kube-api-access-h7svx") pod "7c9b3305-224f-4a35-b878-14d7b23d2c19" (UID: "7c9b3305-224f-4a35-b878-14d7b23d2c19"). InnerVolumeSpecName "kube-api-access-h7svx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:07.396697 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.396650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c9b3305-224f-4a35-b878-14d7b23d2c19" (UID: "7c9b3305-224f-4a35-b878-14d7b23d2c19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:07.495610 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.495573    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7svx\" (UniqueName: \"kubernetes.io/projected/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-api-access-h7svx\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:44:07.495610 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.495604    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c9b3305-224f-4a35-b878-14d7b23d2c19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:44:07.495610 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.495616    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c9b3305-224f-4a35-b878-14d7b23d2c19-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:44:07.618103 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.618064    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerID="bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a" exitCode=0
Sep 17 23:44:07.618513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.618116    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"
Sep 17 23:44:07.618513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.618146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerDied","Data":"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"}
Sep 17 23:44:07.618513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.618209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np" event={"ID":"7c9b3305-224f-4a35-b878-14d7b23d2c19","Type":"ContainerDied","Data":"137d7f0e3f88ef2ab0cedbbd39a0cd9b0f18feec09f636cf283183ea6f7b365a"}
Sep 17 23:44:07.618513 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.618235    2559 scope.go:117] "RemoveContainer" containerID="09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"
Sep 17 23:44:07.627209 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.627190    2559 scope.go:117] "RemoveContainer" containerID="bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"
Sep 17 23:44:07.634695 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.634671    2559 scope.go:117] "RemoveContainer" containerID="09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"
Sep 17 23:44:07.635028 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:44:07.635008    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0\": container with ID starting with 09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0 not found: ID does not exist" containerID="09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"
Sep 17 23:44:07.635101 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.635035    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0"} err="failed to get container status \"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0\": rpc error: code = NotFound desc = could not find container \"09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0\": container with ID starting with 09452146241f5b99914f03312fd910f418776de812dcd0d7a6530b67575889d0 not found: ID does not exist"
Sep 17 23:44:07.635101 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.635050    2559 scope.go:117] "RemoveContainer" containerID="bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"
Sep 17 23:44:07.635273 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:44:07.635257    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a\": container with ID starting with bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a not found: ID does not exist" containerID="bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"
Sep 17 23:44:07.635321 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.635277    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a"} err="failed to get container status \"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a\": rpc error: code = NotFound desc = could not find container \"bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a\": container with ID starting with bb1ab14a403f08bd70353f8178066df2706f48eed8642382b3c00ad532b4950a not found: ID does not exist"
Sep 17 23:44:07.641429 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.641403    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:44:07.648262 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:07.648209    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fc189-predictor-7d7c48b5bf-sm4np"]
Sep 17 23:44:08.469331 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:08.469292    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 23:44:09.649307 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:09.649267    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" path="/var/lib/kubelet/pods/7c9b3305-224f-4a35-b878-14d7b23d2c19/volumes"
Sep 17 23:44:11.617738 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:11.617704    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:44:11.618315 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:11.618288    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:18.470339 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:18.470306    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 17 23:44:21.618804 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:21.618763    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:31.619253 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:31.619210    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:40.020574 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:40.020528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:40.194254 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:40.194218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:41.618789 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:41.618748    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 23:44:51.619464 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:44:51.619429    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:45:13.916559 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:45:13.916515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:15.005591 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:45:15.005551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:24.863611 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:45:24.863570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:33.825066 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:46:33.825019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:33.896159 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:46:33.896116    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:48:44.657037 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:48:44.657001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:50:51.327910 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:50:51.327874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:28.635589 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.635558    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:28.770078 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.770046    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:53:28.770304 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.770281    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container" containerID="cri-o://e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b" gracePeriod=30
Sep 17 23:53:28.770368 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.770322    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kube-rbac-proxy" containerID="cri-o://6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919" gracePeriod=30
Sep 17 23:53:28.773929 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.773887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:28.863194 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:28.863162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:29.086096 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:29.086065    2559 generic.go:356] "Generic (PLEG): container finished" podID="d62ed706-b889-47f5-86dc-2cf982f74298" containerID="6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919" exitCode=2
Sep 17 23:53:29.086227 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:29.086100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerDied","Data":"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"}
Sep 17 23:53:31.415821 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.415801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:53:31.541581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.541505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwm7h\" (UniqueName: \"kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h\") pod \"d62ed706-b889-47f5-86dc-2cf982f74298\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") "
Sep 17 23:53:31.541581 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.541569    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config\") pod \"d62ed706-b889-47f5-86dc-2cf982f74298\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") "
Sep 17 23:53:31.541815 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.541609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls\") pod \"d62ed706-b889-47f5-86dc-2cf982f74298\" (UID: \"d62ed706-b889-47f5-86dc-2cf982f74298\") "
Sep 17 23:53:31.542003 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.541978    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d62ed706-b889-47f5-86dc-2cf982f74298" (UID: "d62ed706-b889-47f5-86dc-2cf982f74298"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:31.543609 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.543589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d62ed706-b889-47f5-86dc-2cf982f74298" (UID: "d62ed706-b889-47f5-86dc-2cf982f74298"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:31.543687 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.543652    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h" (OuterVolumeSpecName: "kube-api-access-pwm7h") pod "d62ed706-b889-47f5-86dc-2cf982f74298" (UID: "d62ed706-b889-47f5-86dc-2cf982f74298"). InnerVolumeSpecName "kube-api-access-pwm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:53:31.642414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.642390    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d62ed706-b889-47f5-86dc-2cf982f74298-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:53:31.642414 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.642412    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d62ed706-b889-47f5-86dc-2cf982f74298-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:53:31.642543 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:31.642423    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwm7h\" (UniqueName: \"kubernetes.io/projected/d62ed706-b889-47f5-86dc-2cf982f74298-kube-api-access-pwm7h\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 17 23:53:32.100781 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.100754    2559 generic.go:356] "Generic (PLEG): container finished" podID="d62ed706-b889-47f5-86dc-2cf982f74298" containerID="e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b" exitCode=0
Sep 17 23:53:32.100923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.100803    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"
Sep 17 23:53:32.100923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.100837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerDied","Data":"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"}
Sep 17 23:53:32.100923 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.100901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l" event={"ID":"d62ed706-b889-47f5-86dc-2cf982f74298","Type":"ContainerDied","Data":"6c71d35467cf5d737139bec4528ebb35f49c04a4f3eddc8ef74439cd1d3dad27"}
Sep 17 23:53:32.101060 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.100939    2559 scope.go:117] "RemoveContainer" containerID="6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"
Sep 17 23:53:32.108622 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.108606    2559 scope.go:117] "RemoveContainer" containerID="e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"
Sep 17 23:53:32.115392 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.115376    2559 scope.go:117] "RemoveContainer" containerID="6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"
Sep 17 23:53:32.115751 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:53:32.115716    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919\": container with ID starting with 6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919 not found: ID does not exist" containerID="6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"
Sep 17 23:53:32.115869 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.115762    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919"} err="failed to get container status \"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919\": rpc error: code = NotFound desc = could not find container \"6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919\": container with ID starting with 6431110000c438e1236a4b8aa91e9c52a184dd0ec50f480c4e97875beb5c5919 not found: ID does not exist"
Sep 17 23:53:32.115869 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.115785    2559 scope.go:117] "RemoveContainer" containerID="e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"
Sep 17 23:53:32.116675 ip-10-0-128-70 kubenswrapper[2559]: E0917 23:53:32.116638    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b\": container with ID starting with e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b not found: ID does not exist" containerID="e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"
Sep 17 23:53:32.116759 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.116736    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b"} err="failed to get container status \"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b\": rpc error: code = NotFound desc = could not find container \"e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b\": container with ID starting with e94d8ec12bb7ff283046afbcef83009cc80b99e8f86539eb8a28d74cf745067b not found: ID does not exist"
Sep 17 23:53:32.117992 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.117975    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:53:32.123785 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:32.123766    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daa97-predictor-7b69c6c9cf-flk4l"]
Sep 17 23:53:33.647947 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:53:33.647913    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" path="/var/lib/kubelet/pods/d62ed706-b889-47f5-86dc-2cf982f74298/volumes"
Sep 17 23:54:28.511085 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:54:28.511044    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 17 23:55:58.756290 ip-10-0-128-70 kubenswrapper[2559]: I0917 23:55:58.756213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:00.143680 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.143634    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29828160-df26v"]
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.143975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.143986    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.143998    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144004    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144018    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144025    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144036    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144042    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144097    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kserve-container"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144106    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144112    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9b3305-224f-4a35-b878-14d7b23d2c19" containerName="kube-rbac-proxy"
Sep 18 00:00:00.146036 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.144118    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d62ed706-b889-47f5-86dc-2cf982f74298" containerName="kserve-container"
Sep 18 00:00:00.146944 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.146926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.149551 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.149525    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-676mz\""
Sep 18 00:00:00.149707 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.149616    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Sep 18 00:00:00.165515 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.165490    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29828160-df26v"]
Sep 18 00:00:00.307737 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.307705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vpj\" (UniqueName: \"kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.307897 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.307767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.409011 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.408898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vpj\" (UniqueName: \"kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.409011 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.409006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.409674 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.409642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.418397 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.418366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vpj\" (UniqueName: \"kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj\") pod \"image-pruner-29828160-df26v\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") " pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.475097 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.475055    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:00.749345 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:00.749314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29828160-df26v"]
Sep 18 00:00:00.751866 ip-10-0-128-70 kubenswrapper[2559]: W0918 00:00:00.751838    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f044c7e_b096_4a0e_bc2f_f910d7da1e7e.slice/crio-d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339 WatchSource:0}: Error finding container d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339: Status 404 returned error can't find the container with id d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339
Sep 18 00:00:01.661616 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:01.661581    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" containerID="362ba0b666760f22b7359385b49aea5859fec822f874a11b933a6e7ff63f63d4" exitCode=0
Sep 18 00:00:01.662025 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:01.661680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29828160-df26v" event={"ID":"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e","Type":"ContainerDied","Data":"362ba0b666760f22b7359385b49aea5859fec822f874a11b933a6e7ff63f63d4"}
Sep 18 00:00:01.662025 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:01.661728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29828160-df26v" event={"ID":"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e","Type":"ContainerStarted","Data":"d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339"}
Sep 18 00:00:02.805368 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:02.805335    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:02.932095 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:02.932066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca\") pod \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") "
Sep 18 00:00:02.932277 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:02.932160    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5vpj\" (UniqueName: \"kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj\") pod \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\" (UID: \"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e\") "
Sep 18 00:00:02.932432 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:02.932409    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca" (OuterVolumeSpecName: "serviceca") pod "8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" (UID: "8f044c7e-b096-4a0e-bc2f-f910d7da1e7e"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:00:02.934220 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:02.934196    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj" (OuterVolumeSpecName: "kube-api-access-h5vpj") pod "8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" (UID: "8f044c7e-b096-4a0e-bc2f-f910d7da1e7e"). InnerVolumeSpecName "kube-api-access-h5vpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:00:03.032824 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:03.032774    2559 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-serviceca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:00:03.032824 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:03.032815    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vpj\" (UniqueName: \"kubernetes.io/projected/8f044c7e-b096-4a0e-bc2f-f910d7da1e7e-kube-api-access-h5vpj\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:00:03.672591 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:03.672548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29828160-df26v" event={"ID":"8f044c7e-b096-4a0e-bc2f-f910d7da1e7e","Type":"ContainerDied","Data":"d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339"}
Sep 18 00:00:03.672792 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:03.672614    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08d1e10467a9cd8245dc12051039ab22f29aca8572c5cb195712ba99d80b339"
Sep 18 00:00:03.672792 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:03.672559    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29828160-df26v"
Sep 18 00:00:59.419372 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.419334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:59.504874 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.504837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:59.511517 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.511488    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 18 00:00:59.511814 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.511769    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container" containerID="cri-o://1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7" gracePeriod=30
Sep 18 00:00:59.511974 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.511864    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kube-rbac-proxy" containerID="cri-o://bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346" gracePeriod=30
Sep 18 00:00:59.563046 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.563013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:59.938650 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.938610    2559 generic.go:356] "Generic (PLEG): container finished" podID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerID="bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346" exitCode=2
Sep 18 00:00:59.938844 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:00:59.938685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerDied","Data":"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"}
Sep 18 00:01:00.502640 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.502602    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wsxk4/must-gather-sxf4c"]
Sep 18 00:01:00.503068 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.502972    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" containerName="image-pruner"
Sep 18 00:01:00.503068 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.502984    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" containerName="image-pruner"
Sep 18 00:01:00.503068 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.503048    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f044c7e-b096-4a0e-bc2f-f910d7da1e7e" containerName="image-pruner"
Sep 18 00:01:00.506118 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.506092    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.508310 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.508276    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wsxk4\"/\"default-dockercfg-tpzw7\""
Sep 18 00:01:00.508484 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.508344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wsxk4\"/\"kube-root-ca.crt\""
Sep 18 00:01:00.509220 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.509198    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wsxk4\"/\"openshift-service-ca.crt\""
Sep 18 00:01:00.516205 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.516158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wsxk4/must-gather-sxf4c"]
Sep 18 00:01:00.518646 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.518605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:00.616777 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.616737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.616995 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.616807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7ck\" (UniqueName: \"kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.717917 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.717882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7ck\" (UniqueName: \"kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.718098 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.718012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.718363 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.718345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.725704 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.725678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7ck\" (UniqueName: \"kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck\") pod \"must-gather-sxf4c\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") " pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:00.817893 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:00.817812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:01.086681 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:01.086638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wsxk4/must-gather-sxf4c"]
Sep 18 00:01:01.089167 ip-10-0-128-70 kubenswrapper[2559]: W0918 00:01:01.089130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7227d3e5_7a30_4b07_9b7e_d09352ed41d3.slice/crio-5017a666880b9a8f90467bd7b3b336ef8e35d889fa28159826bdb7998b29f748 WatchSource:0}: Error finding container 5017a666880b9a8f90467bd7b3b336ef8e35d889fa28159826bdb7998b29f748: Status 404 returned error can't find the container with id 5017a666880b9a8f90467bd7b3b336ef8e35d889fa28159826bdb7998b29f748
Sep 18 00:01:01.091098 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:01.091079    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 00:01:01.947992 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:01.947959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" event={"ID":"7227d3e5-7a30-4b07-9b7e-d09352ed41d3","Type":"ContainerStarted","Data":"5017a666880b9a8f90467bd7b3b336ef8e35d889fa28159826bdb7998b29f748"}
Sep 18 00:01:02.047830 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:02.047773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:03.211839 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.211746    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 18 00:01:03.345358 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.344554    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config\") pod \"d77b7875-e9e9-4a15-8f78-16f94410a879\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") "
Sep 18 00:01:03.345358 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.344649    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls\") pod \"d77b7875-e9e9-4a15-8f78-16f94410a879\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") "
Sep 18 00:01:03.345358 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.344715    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzp8r\" (UniqueName: \"kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r\") pod \"d77b7875-e9e9-4a15-8f78-16f94410a879\" (UID: \"d77b7875-e9e9-4a15-8f78-16f94410a879\") "
Sep 18 00:01:03.345815 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.345784    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d77b7875-e9e9-4a15-8f78-16f94410a879" (UID: "d77b7875-e9e9-4a15-8f78-16f94410a879"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:01:03.349001 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.348958    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d77b7875-e9e9-4a15-8f78-16f94410a879" (UID: "d77b7875-e9e9-4a15-8f78-16f94410a879"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:01:03.349882 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.349846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r" (OuterVolumeSpecName: "kube-api-access-zzp8r") pod "d77b7875-e9e9-4a15-8f78-16f94410a879" (UID: "d77b7875-e9e9-4a15-8f78-16f94410a879"). InnerVolumeSpecName "kube-api-access-zzp8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:01:03.446245 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.446212    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d77b7875-e9e9-4a15-8f78-16f94410a879-proxy-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.446245 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.446242    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzp8r\" (UniqueName: \"kubernetes.io/projected/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-api-access-zzp8r\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.446245 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.446252    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d77b7875-e9e9-4a15-8f78-16f94410a879-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.960183 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.960138    2559 generic.go:356] "Generic (PLEG): container finished" podID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerID="1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7" exitCode=0
Sep 18 00:01:03.960395 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.960196    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"
Sep 18 00:01:03.960395 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.960221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerDied","Data":"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"}
Sep 18 00:01:03.960395 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.960293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n" event={"ID":"d77b7875-e9e9-4a15-8f78-16f94410a879","Type":"ContainerDied","Data":"e5b6a1920f33d457596e45d4bc051aae6de1190ecde6125d45950905cf443d13"}
Sep 18 00:01:03.960395 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.960323    2559 scope.go:117] "RemoveContainer" containerID="bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"
Sep 18 00:01:03.976933 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.976894    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 18 00:01:03.981846 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:03.981809    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57566-predictor-85dbbdfff7-k9s8n"]
Sep 18 00:01:05.649698 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:05.649647    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" path="/var/lib/kubelet/pods/d77b7875-e9e9-4a15-8f78-16f94410a879/volumes"
Sep 18 00:01:06.611917 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:06.611753    2559 scope.go:117] "RemoveContainer" containerID="1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"
Sep 18 00:01:07.550850 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.550811    2559 scope.go:117] "RemoveContainer" containerID="bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"
Sep 18 00:01:07.551262 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:07.551241    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346\": container with ID starting with bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346 not found: ID does not exist" containerID="bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"
Sep 18 00:01:07.551323 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.551274    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346"} err="failed to get container status \"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346\": rpc error: code = NotFound desc = could not find container \"bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346\": container with ID starting with bde4c37f2bd78e34397e03f0571374d427509e8c7da2bee6ae8bb92b0eb57346 not found: ID does not exist"
Sep 18 00:01:07.551323 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.551294    2559 scope.go:117] "RemoveContainer" containerID="1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"
Sep 18 00:01:07.551587 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:07.551568    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7\": container with ID starting with 1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7 not found: ID does not exist" containerID="1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"
Sep 18 00:01:07.551650 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.551599    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7"} err="failed to get container status \"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7\": rpc error: code = NotFound desc = could not find container \"1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7\": container with ID starting with 1b2c36f11421ed4205fdfac00bf3f355f264a55506eda22082845313ff4c81c7 not found: ID does not exist"
Sep 18 00:01:07.986848 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.986810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" event={"ID":"7227d3e5-7a30-4b07-9b7e-d09352ed41d3","Type":"ContainerStarted","Data":"2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19"}
Sep 18 00:01:07.987035 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:07.986860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" event={"ID":"7227d3e5-7a30-4b07-9b7e-d09352ed41d3","Type":"ContainerStarted","Data":"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"}
Sep 18 00:01:08.003374 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:08.003328    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" podStartSLOduration=1.473320513 podStartE2EDuration="8.003312247s" totalImagesPullingTime="6.529991734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:01:08.001611585 +0000 UTC m=+3022.809722735" watchObservedRunningTime="2026-09-18 00:01:08.003312247 +0000 UTC m=+3022.811423375"
Sep 18 00:01:30.099890 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.099850    2559 generic.go:356] "Generic (PLEG): container finished" podID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerID="517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7" exitCode=0
Sep 18 00:01:30.100354 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.099897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" event={"ID":"7227d3e5-7a30-4b07-9b7e-d09352ed41d3","Type":"ContainerDied","Data":"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"}
Sep 18 00:01:30.100354 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.100234    2559 scope.go:117] "RemoveContainer" containerID="517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"
Sep 18 00:01:30.304198 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.304169    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wsxk4_must-gather-sxf4c_7227d3e5-7a30-4b07-9b7e-d09352ed41d3/gather/0.log"
Sep 18 00:01:30.736822 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.736739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:30.869534 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869504    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b94j/must-gather-484zm"]
Sep 18 00:01:30.869860 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869847    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kube-rbac-proxy"
Sep 18 00:01:30.869923 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869862    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kube-rbac-proxy"
Sep 18 00:01:30.869923 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869874    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container"
Sep 18 00:01:30.869923 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869881    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container"
Sep 18 00:01:30.870021 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869938    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kube-rbac-proxy"
Sep 18 00:01:30.870021 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.869947    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77b7875-e9e9-4a15-8f78-16f94410a879" containerName="kserve-container"
Sep 18 00:01:30.872025 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.872004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:30.874371 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.874345    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b94j\"/\"kube-root-ca.crt\""
Sep 18 00:01:30.874371 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.874357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5b94j\"/\"default-dockercfg-zkk62\""
Sep 18 00:01:30.874546 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.874390    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b94j\"/\"openshift-service-ca.crt\""
Sep 18 00:01:30.881204 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.881183    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/must-gather-484zm"]
Sep 18 00:01:30.891266 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.891233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:30.986731 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.986646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3187f64-054c-4f77-bb87-8726d8c85290-must-gather-output\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:30.987022 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.986826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcvb\" (UniqueName: \"kubernetes.io/projected/b3187f64-054c-4f77-bb87-8726d8c85290-kube-api-access-fxcvb\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:30.991338 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:30.991309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:31.087490 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.087452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3187f64-054c-4f77-bb87-8726d8c85290-must-gather-output\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:31.087731 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.087524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcvb\" (UniqueName: \"kubernetes.io/projected/b3187f64-054c-4f77-bb87-8726d8c85290-kube-api-access-fxcvb\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:31.087875 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.087851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3187f64-054c-4f77-bb87-8726d8c85290-must-gather-output\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:31.095174 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.095152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcvb\" (UniqueName: \"kubernetes.io/projected/b3187f64-054c-4f77-bb87-8726d8c85290-kube-api-access-fxcvb\") pod \"must-gather-484zm\" (UID: \"b3187f64-054c-4f77-bb87-8726d8c85290\") " pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:31.181947 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.181905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/must-gather-484zm"
Sep 18 00:01:31.451874 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:31.451771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/must-gather-484zm"]
Sep 18 00:01:31.454385 ip-10-0-128-70 kubenswrapper[2559]: W0918 00:01:31.454360    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3187f64_054c_4f77_bb87_8726d8c85290.slice/crio-60f80a75449b5864a781be35c0eace88bbe73beca371a2034eb1162cd4c50fc8 WatchSource:0}: Error finding container 60f80a75449b5864a781be35c0eace88bbe73beca371a2034eb1162cd4c50fc8: Status 404 returned error can't find the container with id 60f80a75449b5864a781be35c0eace88bbe73beca371a2034eb1162cd4c50fc8
Sep 18 00:01:32.110185 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:32.110149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/must-gather-484zm" event={"ID":"b3187f64-054c-4f77-bb87-8726d8c85290","Type":"ContainerStarted","Data":"60f80a75449b5864a781be35c0eace88bbe73beca371a2034eb1162cd4c50fc8"}
Sep 18 00:01:33.119880 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.119841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/must-gather-484zm" event={"ID":"b3187f64-054c-4f77-bb87-8726d8c85290","Type":"ContainerStarted","Data":"5a31b687f9c8b15734bac555f261f9f40516262054e119ef9446c4966b3eb020"}
Sep 18 00:01:33.120379 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.119895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/must-gather-484zm" event={"ID":"b3187f64-054c-4f77-bb87-8726d8c85290","Type":"ContainerStarted","Data":"2f16b161664f5e251db48bc177eec022e629c5f42c20ef564e3211e8a7f0477f"}
Sep 18 00:01:33.138863 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.138793    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b94j/must-gather-484zm" podStartSLOduration=2.3583525 podStartE2EDuration="3.13876653s" totalImagesPullingTime="780.41403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:01:33.135288103 +0000 UTC m=+3047.943399254" watchObservedRunningTime="2026-09-18 00:01:33.13876653 +0000 UTC m=+3047.946877680"
Sep 18 00:01:33.634864 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.634803    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f2rvg_2b7419d9-a707-42b5-b957-000072e9b3c2/global-pull-secret-syncer/0.log"
Sep 18 00:01:33.777096 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.777066    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log"
Sep 18 00:01:33.886745 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:33.886636    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-70.ec2.internal_2e525bd7c723bc2bc2965a954409bdf5/haproxy/0.log"
Sep 18 00:01:35.725901 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:35.725847    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-wsxk4/must-gather-sxf4c"]
Sep 18 00:01:35.726511 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:35.726153    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-wsxk4/must-gather-sxf4c" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="copy" containerID="cri-o://2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19" gracePeriod=2
Sep 18 00:01:35.728592 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:35.728552    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sxf4c\" is forbidden: User \"system:node:ip-10-0-128-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wsxk4\": no relationship found between node 'ip-10-0-128-70.ec2.internal' and this object" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:35.733152 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:35.732515    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-wsxk4/must-gather-sxf4c"]
Sep 18 00:01:36.131686 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.128973    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wsxk4_must-gather-sxf4c_7227d3e5-7a30-4b07-9b7e-d09352ed41d3/copy/0.log"
Sep 18 00:01:36.138848 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.138763    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:36.141451 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:36.141408    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sxf4c\" is forbidden: User \"system:node:ip-10-0-128-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wsxk4\": no relationship found between node 'ip-10-0-128-70.ec2.internal' and this object" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:36.144309 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.144210    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wsxk4_must-gather-sxf4c_7227d3e5-7a30-4b07-9b7e-d09352ed41d3/copy/0.log"
Sep 18 00:01:36.145487 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.144849    2559 generic.go:356] "Generic (PLEG): container finished" podID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerID="2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19" exitCode=143
Sep 18 00:01:36.145487 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.145010    2559 scope.go:117] "RemoveContainer" containerID="2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19"
Sep 18 00:01:36.145487 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.145162    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:36.147585 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:36.147298    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sxf4c\" is forbidden: User \"system:node:ip-10-0-128-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wsxk4\": no relationship found between node 'ip-10-0-128-70.ec2.internal' and this object" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:36.169888 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.169786    2559 scope.go:117] "RemoveContainer" containerID="517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"
Sep 18 00:01:36.207185 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.207153    2559 scope.go:117] "RemoveContainer" containerID="2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19"
Sep 18 00:01:36.207621 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:36.207577    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19\": container with ID starting with 2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19 not found: ID does not exist" containerID="2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19"
Sep 18 00:01:36.207808 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.207627    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19"} err="failed to get container status \"2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19\": rpc error: code = NotFound desc = could not find container \"2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19\": container with ID starting with 2d33881a0fc68daafcf391d6b9c5659a5483ce5bc00829c166072ff9672d8b19 not found: ID does not exist"
Sep 18 00:01:36.207808 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.207654    2559 scope.go:117] "RemoveContainer" containerID="517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"
Sep 18 00:01:36.207980 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:36.207956    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7\": container with ID starting with 517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7 not found: ID does not exist" containerID="517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"
Sep 18 00:01:36.208052 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.207980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7"} err="failed to get container status \"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7\": rpc error: code = NotFound desc = could not find container \"517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7\": container with ID starting with 517f42e8b8c6b62ce00916402e6339f1a7528c10bdad0e21721b643da2bbebc7 not found: ID does not exist"
Sep 18 00:01:36.244765 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.242446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp7ck\" (UniqueName: \"kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck\") pod \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") "
Sep 18 00:01:36.244765 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.242584    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output\") pod \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\" (UID: \"7227d3e5-7a30-4b07-9b7e-d09352ed41d3\") "
Sep 18 00:01:36.244765 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.244399    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7227d3e5-7a30-4b07-9b7e-d09352ed41d3" (UID: "7227d3e5-7a30-4b07-9b7e-d09352ed41d3"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:01:36.247294 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.247254    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck" (OuterVolumeSpecName: "kube-api-access-vp7ck") pod "7227d3e5-7a30-4b07-9b7e-d09352ed41d3" (UID: "7227d3e5-7a30-4b07-9b7e-d09352ed41d3"). InnerVolumeSpecName "kube-api-access-vp7ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:01:36.344079 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.343993    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-must-gather-output\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:01:36.377991 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.377954    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7ck\" (UniqueName: \"kubernetes.io/projected/7227d3e5-7a30-4b07-9b7e-d09352ed41d3-kube-api-access-vp7ck\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\""
Sep 18 00:01:36.444947 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.443744    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:36.465881 ip-10-0-128-70 kubenswrapper[2559]: E0918 00:01:36.465835    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-sxf4c\" is forbidden: User \"system:node:ip-10-0-128-70.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wsxk4\": no relationship found between node 'ip-10-0-128-70.ec2.internal' and this object" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" pod="openshift-must-gather-wsxk4/must-gather-sxf4c"
Sep 18 00:01:36.516059 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:36.516016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dfdfr_a85598d9-fd95-44dd-b267-ef1921ce3de0/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:37.654269 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:37.653813    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" path="/var/lib/kubelet/pods/7227d3e5-7a30-4b07-9b7e-d09352ed41d3/volumes"
Sep 18 00:01:38.193193 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.193165    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/alertmanager/0.log"
Sep 18 00:01:38.215861 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.215832    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/config-reloader/0.log"
Sep 18 00:01:38.240635 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.240513    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/kube-rbac-proxy-web/0.log"
Sep 18 00:01:38.261530 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.261499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/kube-rbac-proxy/0.log"
Sep 18 00:01:38.285733 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.285700    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/kube-rbac-proxy-metric/0.log"
Sep 18 00:01:38.311785 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.311627    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/prom-label-proxy/0.log"
Sep 18 00:01:38.335898 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.335871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e014c759-1848-4f73-938f-c789d4925569/init-config-reloader/0.log"
Sep 18 00:01:38.402774 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.402725    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xtxxd_fbc7d813-a53e-4f93-8b3e-e396a304a7bb/kube-state-metrics/0.log"
Sep 18 00:01:38.424440 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.424400    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xtxxd_fbc7d813-a53e-4f93-8b3e-e396a304a7bb/kube-rbac-proxy-main/0.log"
Sep 18 00:01:38.447920 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.447826    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-xtxxd_fbc7d813-a53e-4f93-8b3e-e396a304a7bb/kube-rbac-proxy-self/0.log"
Sep 18 00:01:38.479926 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.479885    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-565cbf5f97-g9gjw_647a4953-eca7-4fd6-85d6-ffa639f85641/metrics-server/0.log"
Sep 18 00:01:38.628638 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.628610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6g8m8_80bfd48b-bdb8-4bae-a3d2-2289ee9b6647/node-exporter/0.log"
Sep 18 00:01:38.650416 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.650389    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6g8m8_80bfd48b-bdb8-4bae-a3d2-2289ee9b6647/kube-rbac-proxy/0.log"
Sep 18 00:01:38.671088 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:38.671045    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6g8m8_80bfd48b-bdb8-4bae-a3d2-2289ee9b6647/init-textfile/0.log"
Sep 18 00:01:39.065936 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:39.065904    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-wb7lm_508ac08c-a3ce-40f6-82e5-775f03337d75/prometheus-operator-admission-webhook/0.log"
Sep 18 00:01:39.095921 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:39.095884    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f9b4bc6-tmsmn_0f4bcd70-35a7-4389-8ef1-18cac8a64259/telemeter-client/0.log"
Sep 18 00:01:39.122717 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:39.122686    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f9b4bc6-tmsmn_0f4bcd70-35a7-4389-8ef1-18cac8a64259/reload/0.log"
Sep 18 00:01:39.149247 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:39.149215    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f9b4bc6-tmsmn_0f4bcd70-35a7-4389-8ef1-18cac8a64259/kube-rbac-proxy/0.log"
Sep 18 00:01:41.488540 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:41.488512    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6b6f8d75b-jsx26_0055cfb2-1efa-406e-8532-d706e20ba7bf/console/0.log"
Sep 18 00:01:41.524369 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:41.524338    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-cvxr4_fc5d8827-4b1e-41b8-8b80-e8e9b130a33e/download-server/0.log"
Sep 18 00:01:42.391414 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.391337    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"]
Sep 18 00:01:42.391951 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.391922    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="gather"
Sep 18 00:01:42.391951 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.391952    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="gather"
Sep 18 00:01:42.392120 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.391971    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="copy"
Sep 18 00:01:42.392120 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.391981    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="copy"
Sep 18 00:01:42.392120 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.392071    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="copy"
Sep 18 00:01:42.392120 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.392085    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7227d3e5-7a30-4b07-9b7e-d09352ed41d3" containerName="gather"
Sep 18 00:01:42.399043 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.399015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.404324 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.403806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"]
Sep 18 00:01:42.540655 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.540619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-sys\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.540655 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.540682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-podres\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.541232 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.540768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdxk\" (UniqueName: \"kubernetes.io/projected/c14d750f-ebe4-4187-838b-e03119bf5161-kube-api-access-nzdxk\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.541232 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.540830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-proc\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.541232 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.540857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-lib-modules\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.641702 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-sys\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.641702 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-podres\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.641934 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdxk\" (UniqueName: \"kubernetes.io/projected/c14d750f-ebe4-4187-838b-e03119bf5161-kube-api-access-nzdxk\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.641934 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-proc\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.641934 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-lib-modules\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.642089 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.641980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-lib-modules\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.642089 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.642045    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-sys\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.642196 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.642119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-podres\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.642335 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.642313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c14d750f-ebe4-4187-838b-e03119bf5161-proc\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.651215 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.651184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdxk\" (UniqueName: \"kubernetes.io/projected/c14d750f-ebe4-4187-838b-e03119bf5161-kube-api-access-nzdxk\") pod \"perf-node-gather-daemonset-s7k4r\" (UID: \"c14d750f-ebe4-4187-838b-e03119bf5161\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:42.716152 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:42.716118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:43.007351 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.007145    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"]
Sep 18 00:01:43.010261 ip-10-0-128-70 kubenswrapper[2559]: W0918 00:01:43.010223    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc14d750f_ebe4_4187_838b_e03119bf5161.slice/crio-e214ea77512053130dab34b78f88384471e41234dc959079cfa97d5b9143d4e3 WatchSource:0}: Error finding container e214ea77512053130dab34b78f88384471e41234dc959079cfa97d5b9143d4e3: Status 404 returned error can't find the container with id e214ea77512053130dab34b78f88384471e41234dc959079cfa97d5b9143d4e3
Sep 18 00:01:43.049919 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.049888    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kt4mw_89b6a55a-477e-4e8a-93f3-30d8908ba638/dns/0.log"
Sep 18 00:01:43.083083 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.083053    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kt4mw_89b6a55a-477e-4e8a-93f3-30d8908ba638/kube-rbac-proxy/0.log"
Sep 18 00:01:43.129404 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.129373    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7vljg_5b41259e-6f7f-40e7-9765-c1e01f69c2c8/dns-node-resolver/0.log"
Sep 18 00:01:43.204775 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.204729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r" event={"ID":"c14d750f-ebe4-4187-838b-e03119bf5161","Type":"ContainerStarted","Data":"209d9b9c057559831b5a14ee91cc9044732a507d4369d60fc308840f5d5a5158"}
Sep 18 00:01:43.204967 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.204804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r" event={"ID":"c14d750f-ebe4-4187-838b-e03119bf5161","Type":"ContainerStarted","Data":"e214ea77512053130dab34b78f88384471e41234dc959079cfa97d5b9143d4e3"}
Sep 18 00:01:43.205097 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.205073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:43.224813 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.224735    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r" podStartSLOduration=1.224707746 podStartE2EDuration="1.224707746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:01:43.222531797 +0000 UTC m=+3058.030642949" watchObservedRunningTime="2026-09-18 00:01:43.224707746 +0000 UTC m=+3058.032818899"
Sep 18 00:01:43.645478 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.645449    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29828160-df26v_8f044c7e-b096-4a0e-bc2f-f910d7da1e7e/image-pruner/0.log"
Sep 18 00:01:43.703299 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:43.703269    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xqrv_449a9f0f-970a-48c3-964d-408d9328098a/node-ca/0.log"
Sep 18 00:01:44.824981 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:44.824954    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cfl2d_6be09408-eab2-4ddc-ab4a-c55fafe7dfd1/serve-healthcheck-canary/0.log"
Sep 18 00:01:45.189270 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:45.189235    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-drfk8_67517d31-1446-41a9-92ca-b6869abee076/insights-operator/0.log"
Sep 18 00:01:45.190641 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:45.190609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-drfk8_67517d31-1446-41a9-92ca-b6869abee076/insights-operator/1.log"
Sep 18 00:01:45.375142 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:45.375117    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsdpn_0f5dd514-e15b-4f12-b583-e39c4e9c6bc2/kube-rbac-proxy/0.log"
Sep 18 00:01:45.395469 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:45.395445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsdpn_0f5dd514-e15b-4f12-b583-e39c4e9c6bc2/exporter/0.log"
Sep 18 00:01:45.418117 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:45.418090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsdpn_0f5dd514-e15b-4f12-b583-e39c4e9c6bc2/extractor/0.log"
Sep 18 00:01:47.738856 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:47.738807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-jvsvq_bea51e17-c521-497d-b499-84b634e06cff/manager/0.log"
Sep 18 00:01:47.917175 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:47.917138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-bnxsq_78707091-83de-4ffb-ba94-d93b11ff3ad4/manager/0.log"
Sep 18 00:01:47.950457 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:47.950423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cssbc_1d6854b8-62c8-4a08-a72f-600aa5b57f2c/s3-init/0.log"
Sep 18 00:01:47.966478 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:47.966447    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cssbc_1d6854b8-62c8-4a08-a72f-600aa5b57f2c/download-hf-model/0.log"
Sep 18 00:01:47.992354 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:47.992266    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-f2qc4_bc44ba4c-4048-4f38-b017-1f1a5c2ddda3/seaweedfs/0.log"
Sep 18 00:01:49.223738 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:49.223708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-s7k4r"
Sep 18 00:01:52.341831 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:52.341792    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-gpvx7_95e5e3d6-dbcb-4de2-a4de-a26a8c94c301/kube-storage-version-migrator-operator/1.log"
Sep 18 00:01:52.342696 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:52.342654    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-gpvx7_95e5e3d6-dbcb-4de2-a4de-a26a8c94c301/kube-storage-version-migrator-operator/0.log"
Sep 18 00:01:53.297336 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.297307    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:01:53.317450 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.317420    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/egress-router-binary-copy/0.log"
Sep 18 00:01:53.335616 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.335563    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/cni-plugins/0.log"
Sep 18 00:01:53.354447 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.354399    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/bond-cni-plugin/0.log"
Sep 18 00:01:53.373705 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.373635    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/routeoverride-cni/0.log"
Sep 18 00:01:53.392576 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.392538    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/whereabouts-cni-bincopy/0.log"
Sep 18 00:01:53.412026 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.411997    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9lhnq_697bafa8-05d3-48dc-809c-e37e84377208/whereabouts-cni/0.log"
Sep 18 00:01:53.770357 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.770321    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-smjgv_99bde14c-570c-4e9c-9224-62729c4a6ebd/kube-multus/0.log"
Sep 18 00:01:53.909157 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.909105    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zmndj_21037b83-b73d-4375-9b84-b018549bbc05/network-metrics-daemon/0.log"
Sep 18 00:01:53.926884 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:53.926856    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zmndj_21037b83-b73d-4375-9b84-b018549bbc05/kube-rbac-proxy/0.log"
Sep 18 00:01:54.614799 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.614761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/ovn-controller/0.log"
Sep 18 00:01:54.630247 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.630220    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/ovn-acl-logging/0.log"
Sep 18 00:01:54.651184 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.651155    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/kube-rbac-proxy-node/0.log"
Sep 18 00:01:54.674050 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.674016    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:01:54.691854 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.691827    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/northd/0.log"
Sep 18 00:01:54.712144 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.712105    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/nbdb/0.log"
Sep 18 00:01:54.734771 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.734739    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/sbdb/0.log"
Sep 18 00:01:54.774970 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:54.774938    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r92zk_0f72f03c-3028-4e21-b64e-0c2fe24e4156/ovnkube-controller/0.log"
Sep 18 00:01:56.177570 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:56.177542    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9pjl4_997de6e1-d5a5-4a6e-af1f-7492043ddbc6/network-check-target-container/0.log"
Sep 18 00:01:57.141302 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:57.141270    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5mswh_aa135e44-b6c0-42f5-b177-fb9a96c2f35a/iptables-alerter/0.log"
Sep 18 00:01:57.843092 ip-10-0-128-70 kubenswrapper[2559]: I0918 00:01:57.843061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-t9n76_633cba43-1482-404e-9ae7-556d35e07bff/tuned/0.log"