Jul 07 14:54:45.340885 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 14:54:45.340894 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 14:54:45.340901 ip-10-0-128-215 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 14:54:45.341228 ip-10-0-128-215 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 14:54:55.571994 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 14:54:55.572017 ip-10-0-128-215 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 447201e0763740409c24886776caa228 --
Jul 07 14:56:58.232298 ip-10-0-128-215 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 14:56:58.637317 ip-10-0-128-215 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:56:58.637317 ip-10-0-128-215 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 14:56:58.637317 ip-10-0-128-215 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:56:58.637317 ip-10-0-128-215 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 14:56:58.637317 ip-10-0-128-215 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:56:58.638026 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.637299    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 14:56:58.641888 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641868    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.641888 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641887    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641891    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641894    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641897    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641900    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641903    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641906    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641908    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641911    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641913    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641916    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641919    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641921    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641924    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641926    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641929    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641948    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641951    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641954    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.641971 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.641957    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642015    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642066    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642071    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642077    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642081    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.642667 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642086    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.643176 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.642091    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643182    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643188    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643195    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643199    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643204    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643209    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643213    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643218    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643223    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643228    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.643231 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643236    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643239    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643261    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643373    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643380    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643383    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643386    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643389    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643392    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643395    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643398    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643400    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643403    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643405    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643408    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643411    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643413    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643416    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643419    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643421    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.643509 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643428    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643433    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643438    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643441    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643445    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643450    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643453    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643455    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643458    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643460    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643463    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643466    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643468    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643471    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643475    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643478    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643481    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643484    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643487    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.644006 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643490    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643492    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643497    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643499    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643502    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643504    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643507    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643509    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643512    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643515    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643518    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643520    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643523    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643526    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643528    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643532    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643534    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643537    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643539    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643542    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.644458 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643545    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.643548    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644350    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644358    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644361    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644365    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644368    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644371    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644373    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644376    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644379    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644382    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644385    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644388    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644391    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644394    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644396    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644399    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644402    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644404    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.644944 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644407    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644410    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644412    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644415    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644417    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644420    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644422    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644425    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644428    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644431    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644433    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644435    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644438    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644440    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644442    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644445    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644447    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644450    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644452    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.645433 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644454    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644457    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644460    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644463    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644466    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644468    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644471    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644474    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644477    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644479    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644482    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644484    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644487    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644490    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644493    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644495    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644498    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644500    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644503    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644505    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.645896 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644507    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644510    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644512    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644517    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644520    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644523    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644526    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644529    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644532    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644536    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644540    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644542    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644546    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644548    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644552    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644554    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644557    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644559    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644562    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.646402 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644565    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644567    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644570    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644572    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644575    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644577    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644580    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644582    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644585    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644587    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644590    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644593    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644597    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644600    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644603    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644606    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644609    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644611    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644614    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.646867 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644616    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644618    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644621    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.644623    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645030    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645035    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645039    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645042    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645046    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645049    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645052    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645054    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645057    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645059    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645062    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645064    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645067    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645069    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645072    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645074    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.647372 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645077    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645079    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645081    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645084    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645086    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645089    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645091    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645093    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645096    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645098    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645100    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645103    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645105    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645108    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645110    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645113    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645116    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645118    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645121    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645124    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.647879 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645126    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645129    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645132    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645135    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645138    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645140    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645144    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645147    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645149    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645151    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645154    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645156    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645159    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645161    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645163    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645166    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645168    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645171    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645173    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645176    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.648378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645178    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645181    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645183    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645185    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645188    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645190    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645193    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645195    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645198    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645200    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645203    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645205    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645208    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645210    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645213    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645216    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645219    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645221    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645223    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645226    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.648887 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645228    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645230    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645233    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645235    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645237    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645240    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645243    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645247    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645249    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645252    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645255    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645259    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645261    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645263    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645267    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645271    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645274    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645276    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645279    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.649368 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645282    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645286    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645289    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.645291    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645366    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645374    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645388    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645399    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645404    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645407    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645411    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645415    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645418    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645421    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645424    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645427    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645431    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645436    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645440    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645443    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645445    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645448    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645453    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 07 14:56:58.649815 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645456    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645459    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645461    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645465    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645468    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645471    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645474    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645477    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645480    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645483    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645486    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645489    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645492    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645497    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645500    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645503    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645506    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645509    2575 flags.go:64] FLAG: --enable-server="true"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645513    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645517    2575 flags.go:64] FLAG: --event-burst="100"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645520    2575 flags.go:64] FLAG: --event-qps="50"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645523    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645526    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645528    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 07 14:56:58.650411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645532    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645535    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645539    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645542    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645544    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645547    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645550    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645553    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645555    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645558    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645561    2575 flags.go:64] FLAG: --feature-gates=""
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645565    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645568    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645571    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645574    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645577    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645579    2575 flags.go:64] FLAG: --help="false"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645582    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-215.ec2.internal"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645587    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645590    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645593    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645596    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645599    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645602    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 14:56:58.650988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645604    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645607    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645610    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645613    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645616    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645619    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645622    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645624    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645627    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645630    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645634    2575 flags.go:64] FLAG: --lock-file=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645637    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645639    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645642    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645647    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645650    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645653    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645656    2575 flags.go:64] FLAG: --logging-format="text"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645658    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645662    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645665    2575 flags.go:64] FLAG: --manifest-url=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645667    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645671    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645674    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645678    2575 flags.go:64] FLAG: --max-pods="110"
Jul 07 14:56:58.651588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645681    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645685    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645688    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645691    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645694    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645697    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645699    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645708    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645710    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645713    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645716    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645722    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645728    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645730    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645733    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645736    2575 flags.go:64] FLAG: --port="10250"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645739    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645743    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0794b634ed2dd90b9"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645746    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645749    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645752    2575 flags.go:64] FLAG: --register-node="true"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645755    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645759    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645761    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 07 14:56:58.652203 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645764    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645767    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645770    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645773    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645776    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645779    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645782    2575 flags.go:64] FLAG: --runonce="false"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645784    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645787    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645790    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645794    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645797    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645800    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645803    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645806    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645809    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645811    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645814    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645817    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645820    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645823    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645825    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645830    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645833    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645836    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 14:56:58.653197 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645842    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645844    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645847    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645850    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645852    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645855    2575 flags.go:64] FLAG: --v="2"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645859    2575 flags.go:64] FLAG: --version="false"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645863    2575 flags.go:64] FLAG: --vmodule=""
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645866    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.645869    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646025    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646030    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646033    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646036    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646039    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646041    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646044    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646048    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646051    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646053    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646055    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646058    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.654265 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646060    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646063    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646066    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646068    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646071    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646073    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646077    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646080    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646082    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646084    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646089    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646091    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646094    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646097    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646099    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646101    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646104    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646106    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646109    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.655110 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646111    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646113    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646116    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646118    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646121    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646123    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646126    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646128    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646133    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646136    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646144    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646149    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646152    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646154    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646157    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646160    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646162    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646166    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646170    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646173    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.655573 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646178    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646181    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646183    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646187    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646190    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646192    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646195    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646197    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646199    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646202    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646204    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646207    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646209    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646212    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646214    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646218    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646221    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646224    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646227    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646229    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.656606 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646232    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646235    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646237    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646240    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646243    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646245    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646247    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646250    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646252    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646255    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646257    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646260    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646262    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646267    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646270    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646274    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646277    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646280    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646283    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.657343 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646285    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646288    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646291    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646294    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646297    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646299    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646302    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646304    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.646307    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.646857    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.654800    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.654825    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.655580    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.655592    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655712    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.657903 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655721    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655727    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655732    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655736    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655741    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655745    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655749    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655754    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655758    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655763    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655767    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655771    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655775    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655779    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655783    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655787    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655792    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655796    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655802    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655808    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.658501 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655813    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655817    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655822    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655827    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655831    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655835    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655840    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655844    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655850    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655854    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655859    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655863    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655870    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655876    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655881    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655885    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655890    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655894    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655898    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.659013 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655903    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655907    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655911    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655916    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655920    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655924    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655928    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655950    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655954    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655958    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655962    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655966    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655969    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655973    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655976    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655980    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655984    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655988    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655992    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655995    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.659661 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.655999    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656004    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656009    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656012    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656016    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656020    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656024    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656028    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656031    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656035    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656038    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656042    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656046    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656049    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656053    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656057    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656061    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656065    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656071    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.660377 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656075    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656080    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656084    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656088    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656092    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656097    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656101    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656105    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656109    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656113    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656117    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656121    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656126    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656131    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656135    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656139    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656144    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656149    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656153    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.660978 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656157    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.656165    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656371    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656379    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656383    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656387    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656391    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656395    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656400    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656404    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656408    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656412    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656416    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656420    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656424    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:56:58.661472 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656428    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656432    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656436    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656441    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656444    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656450    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656454    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656458    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656461    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656466    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656470    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656475    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656479    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656484    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656488    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656493    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656497    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656502    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656509    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656513    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:56:58.661874 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656517    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656521    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656525    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656530    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656534    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656538    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656542    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656548    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656553    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656558    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656562    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656566    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656571    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656575    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656579    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656583    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656587    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656591    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656595    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:56:58.662389 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656599    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656603    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656607    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656611    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656615    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656619    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656623    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656627    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656630    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656636    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656640    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656644    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656649    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656653    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656657    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656661    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656665    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656669    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656673    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656677    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:56:58.662911 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656681    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656687    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656692    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656697    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656702    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656706    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656710    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656714    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656718    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656722    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656727    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656731    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656735    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656739    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656743    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656748    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656752    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656756    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656760    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:56:58.663507 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656764    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656769    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656773    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656786    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656791    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656795    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656799    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:58.656802    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.656811    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.657025    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.660769    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.662632    2575 server.go:1019] "Starting client certificate rotation"
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.662747    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:56:58.664069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.662784    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:56:58.665517 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.665497    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:56:58.667119 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.667088    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:56:58.683072 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.683042    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 07 14:56:58.685353 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.685335    2575 log.go:25] "Validated CRI v1 image API"
Jul 07 14:56:58.686088 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.686072    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 14:56:58.688494 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.688474    2575 fs.go:135] Filesystem UUIDs: map[146ca09d-8ef5-4e0a-ac8f-ee202edc181a:/dev/nvme0n1p3 5d9f1de8-2915-4a4c-9f31-93978a616015:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 07 14:56:58.688570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.688493    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 14:56:58.694366 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.694235    2575 manager.go:216] Machine: {Timestamp:2026-07-07 14:56:58.692925045 +0000 UTC m=+0.339464139 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102017 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27224c8311178fc8b160df484a7cd5 SystemUUID:ec27224c-8311-178f-c8b1-60df484a7cd5 BootID:447201e0-7637-4040-9c24-886776caa228 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:6f:c6:c7:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:6f:c6:c7:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:18:c1:75:14:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 14:56:58.694366 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.694353    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 14:56:58.694520 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.694449    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 14:56:58.694520 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.694511    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:56:58.697860 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.697828    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 14:56:58.698057 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.697863    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-215.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 14:56:58.698714 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.698700    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 14:56:58.698778 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.698720    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 14:56:58.698778 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.698731    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:56:58.698778 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.698752    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:56:58.700222 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.700208    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 14:56:58.703449 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703434    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:56:58.703584 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703572    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 14:56:58.703639 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703612    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 14:56:58.703639 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703627    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 14:56:58.703743 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703654    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 14:56:58.703743 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703669    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 07 14:56:58.703743 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.703691    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 14:56:58.704225 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.704210    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:56:58.704293 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.704238    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:56:58.707365 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.707350    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 14:56:58.709127 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.709114    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 14:56:58.709170 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.709145    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 14:56:58.711265 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711242    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711274    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711285    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711296    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711310    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711319    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711332    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711340    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 14:56:58.711352 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711353    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 14:56:58.711697 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711367    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 14:56:58.711697 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711382    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 14:56:58.711697 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.711392    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 14:56:58.712492 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.712476    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 14:56:58.712555 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.712497    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 14:56:58.715952 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.715925    2575 server.go:1268] "Started kubelet"
Jul 07 14:56:58.716052 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.716018    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-215.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 14:56:58.716132 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.716015    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 14:56:58.716132 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.716035    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 14:56:58.716229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.716142    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 14:56:58.716344 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.716308    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 14:56:58.716452 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.716426    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 14:56:58.716986 ip-10-0-128-215 systemd[1]: Started Kubernetes Kubelet.
Jul 07 14:56:58.717805 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.717788    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 14:56:58.718846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.718832    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 14:56:58.721633 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.721613    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 14:56:58.724615 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.723765    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-215.ec2.internal.18c009cab8206b10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-215.ec2.internal,UID:ip-10-0-128-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-215.ec2.internal,},FirstTimestamp:2026-07-07 14:56:58.715892496 +0000 UTC m=+0.362431591,LastTimestamp:2026-07-07 14:56:58.715892496 +0000 UTC m=+0.362431591,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-215.ec2.internal,}"
Jul 07 14:56:58.725665 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.725646    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 14:56:58.725752 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.725648    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 14:56:58.725846 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.725770    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:58.726331 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726314    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 14:56:58.726331 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726323    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 14:56:58.726473 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726339    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 14:56:58.726473 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726437    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 14:56:58.726473 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726450    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 14:56:58.726976 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.726951    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzm69"
Jul 07 14:56:58.727229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727203    2575 factory.go:55] Registering systemd factory
Jul 07 14:56:58.727324 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727249    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 07 14:56:58.727500 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727432    2575 factory.go:153] Registering CRI-O factory
Jul 07 14:56:58.727500 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727447    2575 factory.go:223] Registration of the crio container factory successfully
Jul 07 14:56:58.727586 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727501    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 14:56:58.727586 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727525    2575 factory.go:103] Registering Raw factory
Jul 07 14:56:58.727586 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.727539    2575 manager.go:1196] Started watching for new ooms in manager
Jul 07 14:56:58.728150 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.728122    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 14:56:58.728241 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.728224    2575 manager.go:349] Starting recovery of all containers
Jul 07 14:56:58.728372 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.728343    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 14:56:58.736308 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.736267    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzm69"
Jul 07 14:56:58.741588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.741568    2575 manager.go:354] Recovery completed
Jul 07 14:56:58.746147 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.746127    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:58.748001 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.747983    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:58.748066 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748014    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:58.748066 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748024    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:58.748591 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748577    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 14:56:58.748591 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748591    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 14:56:58.748725 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748592    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:58.748725 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.748610    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:56:58.749336 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749321    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:58.749399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749350    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:58.749399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749360    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:58.749786 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749774    2575 policy_none.go:49] "None policy: Start"
Jul 07 14:56:58.749843 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749792    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 14:56:58.749843 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.749807    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 14:56:58.751094 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.751080    2575 policy_none.go:47] "Start"
Jul 07 14:56:58.780518 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.780501    2575 manager.go:337] "Starting Device Plugin manager"
Jul 07 14:56:58.780615 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.780535    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 14:56:58.780615 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.780545    2575 server.go:87] "Starting device plugin registration server"
Jul 07 14:56:58.780795 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.780784    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 14:56:58.780843 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.780798    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 14:56:58.780951 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.780903    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 14:56:58.781029 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.781017    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 14:56:58.781066 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.781030    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 14:56:58.781508 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.781491    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 14:56:58.781594 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.781539    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:58.881211 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.881173    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:58.882454 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.882176    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:58.882454 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.882209    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:58.882454 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.882219    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:58.882454 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.882245    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-215.ec2.internal"
Jul 07 14:56:58.890448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.890400    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-215.ec2.internal"
Jul 07 14:56:58.890448 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.890424    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-215.ec2.internal\": node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:58.907543 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.907514    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:58.912517 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.912479    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 14:56:58.913865 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.913846    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 14:56:58.913916 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.913896    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 14:56:58.913973 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.913921    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 14:56:58.914026 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:58.913983    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 14:56:58.917701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:58.917675    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:56:59.008117 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.008083    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.014248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.014209    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"]
Jul 07 14:56:59.014328 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.014294    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:59.015991 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.015974    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:59.016051 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.016004    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:59.016051 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.016016    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:59.017441 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.017429    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:59.017546 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.017532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.017600 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.017576    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:59.018474 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018454    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:59.018570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018484    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:59.018570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018516    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:59.018570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018525    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:59.018570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018561    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:59.018570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.018573    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:59.019809 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.019796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.019865 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.019819    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:56:59.020693 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.020678    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:56:59.020693 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.020698    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:56:59.020840 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.020719    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:56:59.028121 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.028102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.028202 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.028129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04dfa3a43b9c6e6928c1a69c4428406e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"04dfa3a43b9c6e6928c1a69c4428406e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.028202 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.028150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.042815 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.042787    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-215.ec2.internal\" not found" node="ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.047470 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.047450    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-215.ec2.internal\" not found" node="ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.108379 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.108344    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.128580 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04dfa3a43b9c6e6928c1a69c4428406e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"04dfa3a43b9c6e6928c1a69c4428406e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.128673 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.128673 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128604    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.128673 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128644    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04dfa3a43b9c6e6928c1a69c4428406e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"04dfa3a43b9c6e6928c1a69c4428406e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.128801 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128644    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.128801 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.128703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a84977041e1d635d44b56940b873b525-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"a84977041e1d635d44b56940b873b525\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.209087 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.208994    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.313793 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.313737    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.346730 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.346679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.351253 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.351198    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:56:59.414669 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.414634    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.519476 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.519382    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.526694 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.526666    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:56:59.586121 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.586085    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:56:59.620292 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.620248    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.661733 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.661693    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 14:56:59.662318 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.661854    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:56:59.662318 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.661892    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:56:59.662318 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.661912    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:56:59.720699 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.720658    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.739822 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.739785    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:51:58 +0000 UTC" deadline="2027-12-19 23:10:09.050957635 +0000 UTC"
Jul 07 14:56:59.739822 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.739817    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12728h13m9.311144408s"
Jul 07 14:56:59.821516 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.821441    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.922603 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:56:59.922571    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:56:59.948158 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:59.948127    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dfa3a43b9c6e6928c1a69c4428406e.slice/crio-9bb84fb4cee2b76fed29ea3d38bc717f5b9df827bddcd80dc519d87c43e603d5 WatchSource:0}: Error finding container 9bb84fb4cee2b76fed29ea3d38bc717f5b9df827bddcd80dc519d87c43e603d5: Status 404 returned error can't find the container with id 9bb84fb4cee2b76fed29ea3d38bc717f5b9df827bddcd80dc519d87c43e603d5
Jul 07 14:56:59.948485 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:56:59.948461    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84977041e1d635d44b56940b873b525.slice/crio-19d0459d3acf9536611d35a8c70d337e4df4ad6f5b088abefca8f4de90792b51 WatchSource:0}: Error finding container 19d0459d3acf9536611d35a8c70d337e4df4ad6f5b088abefca8f4de90792b51: Status 404 returned error can't find the container with id 19d0459d3acf9536611d35a8c70d337e4df4ad6f5b088abefca8f4de90792b51
Jul 07 14:56:59.955011 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:56:59.954973    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:57:00.023174 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.023126    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:57:00.125253 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.125172    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:57:00.225816 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.225782    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 07 14:57:00.296370 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.296341    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:00.326561 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.326531    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 07 14:57:00.343250 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.343222    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:57:00.344469 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.344450    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 07 14:57:00.354215 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.354194    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:57:00.704437 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.704401    2575 apiserver.go:52] "Watching apiserver"
Jul 07 14:57:00.711395 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.711193    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 14:57:00.712512 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.712484    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-szttf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal","openshift-multus/multus-additional-cni-plugins-kzczk","openshift-multus/network-metrics-daemon-gqtvk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n","openshift-cluster-node-tuning-operator/tuned-qfn9v","openshift-multus/multus-6fgxt","openshift-network-diagnostics/network-check-target-rbkcv","openshift-network-operator/iptables-alerter-2q8zf","openshift-ovn-kubernetes/ovnkube-node-ftfml","kube-system/global-pull-secret-syncer-lt87x","kube-system/konnectivity-agent-zrkcb","kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal","openshift-dns/node-resolver-2rvd7"]
Jul 07 14:57:00.714308 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.714285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.714415 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.714369    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:00.715379 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.715360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.716723 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.716700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.718070 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.718049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.718762 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.718745    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.719007 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.718990    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9s2t\""
Jul 07 14:57:00.719074 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.719243 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719219    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 14:57:00.719335 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719319    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 14:57:00.719430 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6wj7v\""
Jul 07 14:57:00.719574 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719559    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.719663 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719648    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 14:57:00.719862 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719850    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.719926 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.719912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.720097 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.720082    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 14:57:00.720607 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.720353    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 14:57:00.720607 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.720406    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5dwqx\""
Jul 07 14:57:00.720749 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.720717    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.724229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.723117    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.724229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.723276    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.724229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.723284    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.724229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.723378    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w9sqg\""
Jul 07 14:57:00.724229 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.723482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzhpl\""
Jul 07 14:57:00.725217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.724830    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.725217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.724834    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 14:57:00.725217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.724976    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:00.725217 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.725061    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:00.726706 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.726196    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 14:57:00.726795 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.726757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.727990 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.727962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.729039 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.729018    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.729276 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.729254    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z2mgc\""
Jul 07 14:57:00.729355 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.729279    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.729795 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.729774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.729892 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.729854    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:00.730141 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730127    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.730407 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730391    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 14:57:00.730722 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730700    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 14:57:00.730853 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730741    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 14:57:00.731029 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 14:57:00.731029 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.730981    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 14:57:00.731207 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.731183    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.731742 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.731723    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bml5m\""
Jul 07 14:57:00.731994 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.731774    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.732591 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.732570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.735217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735017    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 14:57:00.735217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735031    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 14:57:00.735522 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735504    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 14:57:00.735628 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735610    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zf26w\""
Jul 07 14:57:00.735722 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735707    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 14:57:00.735722 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.735685    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxnk7\""
Jul 07 14:57:00.740744 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-sys-fs\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.740833 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-socket-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.740833 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-slash\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.740908 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-sys\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.740908 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741060 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740909    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-run\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741060 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r85c\" (UniqueName: \"kubernetes.io/projected/740fa349-e5f2-4369-935e-8ee9ff32fb74-kube-api-access-5r85c\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.741060 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.740996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-device-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.741060 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2bf\" (UniqueName: \"kubernetes.io/projected/70524e5b-7ba0-4696-b5c1-76288ad7c029-kube-api-access-wz2bf\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741060 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-os-release\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-multus\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/740fa349-e5f2-4369-935e-8ee9ff32fb74-host\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-cni-binary-copy\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-host\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741248 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-tuned\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjd5l\" (UniqueName: \"kubernetes.io/projected/48462f54-2be0-43f6-9c9d-d17214830d69-kube-api-access-cjd5l\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741311    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-var-lib-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-system-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysconfig\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741453 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52846\" (UniqueName: \"kubernetes.io/projected/b6497f60-17b2-4e9c-bd83-17c0f13f3956-kube-api-access-52846\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741728 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741728 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-script-lib\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741728 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-daemon-config\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.741728 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-tmp\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.741914 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfd9\" (UniqueName: \"kubernetes.io/projected/42e7abb5-296e-4142-9ed6-0859b5613eb3-kube-api-access-fsfd9\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.741914 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-node-log\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741914 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-log-socket\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.741914 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-kubelet\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.741990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-cnibin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzfn\" (UniqueName: \"kubernetes.io/projected/a5395966-5e5b-4dca-a539-1281a2cc96df-kube-api-access-5vzfn\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-lib-modules\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.742111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/740fa349-e5f2-4369-935e-8ee9ff32fb74-serviceca\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.742247 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-bin\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742247 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-kubernetes\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.742300 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.742300 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-os-release\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742355 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742385 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-systemd\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.742416 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-netns\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742416 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-netns\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742471 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-conf-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742500 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-etc-kubernetes\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742527 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742559 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-registration-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.742559 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-systemd-units\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742648 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-ovn\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742648 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-kubelet\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742712 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742712 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-k8s-cni-cncf-io\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742768 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-bin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742768 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-hostroot\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.742825 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-cnibin\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742825 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742825 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-env-overrides\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovn-node-metrics-cert\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-systemd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-etc-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljw98\" (UniqueName: \"kubernetes.io/projected/1aea3089-4245-4500-80c3-3fc0151b6c96-kube-api-access-ljw98\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-modprobe-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.742921 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-var-lib-kubelet\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.742930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42e7abb5-296e-4142-9ed6-0859b5613eb3-iptables-alerter-script\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e7abb5-296e-4142-9ed6-0859b5613eb3-host-slash\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqwg\" (UniqueName: \"kubernetes.io/projected/cd05c438-977e-42f1-ac3a-10605f8f2d41-kube-api-access-qwqwg\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-socket-dir-parent\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-multus-certs\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-conf\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-netd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-config\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743090    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:57:00.743291 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.743150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-system-cni-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.771261 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.771232    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cn56c"
Jul 07 14:57:00.784009 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.782534    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cn56c"
Jul 07 14:57:00.828150 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.828115    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 14:57:00.843450 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843423    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:00.843632 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/740fa349-e5f2-4369-935e-8ee9ff32fb74-serviceca\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.843632 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-bin\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.843632 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-kubernetes\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.843632 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-os-release\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-bin\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-systemd\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-netns\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-netns\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-conf-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-etc-kubernetes\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843800    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.843846 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-registration-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843851    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-kubernetes\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-systemd-units\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-ovn\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-netns\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-kubelet\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843957    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-ovn\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.843971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-k8s-cni-cncf-io\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844013    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-conf-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844029    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/740fa349-e5f2-4369-935e-8ee9ff32fb74-serviceca\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844048    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-etc-kubernetes\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-bin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844060    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-bin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-hostroot\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-cnibin\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844130    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-systemd\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.844448 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-env-overrides\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovn-node-metrics-cert\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-systemd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844232    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-etc-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljw98\" (UniqueName: \"kubernetes.io/projected/1aea3089-4245-4500-80c3-3fc0151b6c96-kube-api-access-ljw98\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-modprobe-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-var-lib-kubelet\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42e7abb5-296e-4142-9ed6-0859b5613eb3-iptables-alerter-script\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e7abb5-296e-4142-9ed6-0859b5613eb3-host-slash\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844431    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-env-overrides\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844464    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-netns\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqwg\" (UniqueName: \"kubernetes.io/projected/cd05c438-977e-42f1-ac3a-10605f8f2d41-kube-api-access-qwqwg\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844490    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-hostroot\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-socket-dir-parent\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-multus-certs\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.845305 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-conf\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844569    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58e1c6b8-f561-4076-9157-299ed47c3827-agent-certs\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844585    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58e1c6b8-f561-4076-9157-299ed47c3827-konnectivity-ca\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-netd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844642    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-os-release\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-config\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-system-cni-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844675    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-registration-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-kubelet-config\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844702    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-systemd-units\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74cf5a19-c3f7-414c-963f-ac9a91e6f235-hosts-file\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844726    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-cnibin\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/74cf5a19-c3f7-414c-963f-ac9a91e6f235-kube-api-access-pgf8q\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844756    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844771    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-sys-fs\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.846146 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844776    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42e7abb5-296e-4142-9ed6-0859b5613eb3-host-slash\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-etc-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-socket-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844819    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-kubelet\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-slash\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-sys\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844873    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-modprobe-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-run\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844904    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-var-lib-kubelet\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r85c\" (UniqueName: \"kubernetes.io/projected/740fa349-e5f2-4369-935e-8ee9ff32fb74-kube-api-access-5r85c\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.844977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cf5a19-c3f7-414c-963f-ac9a91e6f235-tmp-dir\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-device-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2bf\" (UniqueName: \"kubernetes.io/projected/70524e5b-7ba0-4696-b5c1-76288ad7c029-kube-api-access-wz2bf\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845073    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-os-release\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-multus\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.846889 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845183    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aea3089-4245-4500-80c3-3fc0151b6c96-system-cni-dir\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/740fa349-e5f2-4369-935e-8ee9ff32fb74-host\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.845225    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-cni-binary-copy\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845237    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-slash\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.845307    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:01.345268897 +0000 UTC m=+2.991807998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-host\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-tuned\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjd5l\" (UniqueName: \"kubernetes.io/projected/48462f54-2be0-43f6-9c9d-d17214830d69-kube-api-access-cjd5l\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-var-lib-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-system-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845492    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysconfig\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.847498 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845535    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-cni-netd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52846\" (UniqueName: \"kubernetes.io/projected/b6497f60-17b2-4e9c-bd83-17c0f13f3956-kube-api-access-52846\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/740fa349-e5f2-4369-935e-8ee9ff32fb74-host\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845636    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysconfig\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845648    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-config\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-sys\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845708    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42e7abb5-296e-4142-9ed6-0859b5613eb3-iptables-alerter-script\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845728    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-k8s-cni-cncf-io\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-script-lib\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845735    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aea3089-4245-4500-80c3-3fc0151b6c96-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845742    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-host\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845737    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-d\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845753    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-var-lib-cni-multus\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-daemon-config\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845791    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.848093 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845328    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-run\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845807    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-sys-fs\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845830    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-socket-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845888    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-var-lib-openvswitch\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845904    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-host-run-multus-certs\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-tmp\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfd9\" (UniqueName: \"kubernetes.io/projected/42e7abb5-296e-4142-9ed6-0859b5613eb3-kube-api-access-fsfd9\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-node-log\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-log-socket\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-dbus\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-kubelet\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846182    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-os-release\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846202    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-sysctl-conf\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846255    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-system-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-cnibin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzfn\" (UniqueName: \"kubernetes.io/projected/a5395966-5e5b-4dca-a539-1281a2cc96df-kube-api-access-5vzfn\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846293    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-cnibin\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.848699 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-lib-modules\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovnkube-script-lib\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846384    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-cni-binary-copy\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846392    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-daemon-config\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846410    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-cni-dir\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846435    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-node-log\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846445    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-kubelet\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846459    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6497f60-17b2-4e9c-bd83-17c0f13f3956-lib-modules\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846473    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-log-socket\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846527    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5395966-5e5b-4dca-a539-1281a2cc96df-multus-socket-dir-parent\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846604    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-host-run-ovn-kubernetes\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.845495    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70524e5b-7ba0-4696-b5c1-76288ad7c029-run-systemd\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.846659    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48462f54-2be0-43f6-9c9d-d17214830d69-device-dir\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.849401 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.849285    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-tmp\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.849847 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.849455    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70524e5b-7ba0-4696-b5c1-76288ad7c029-ovn-node-metrics-cert\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.851363 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.851338    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b6497f60-17b2-4e9c-bd83-17c0f13f3956-etc-tuned\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.858781 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.858755    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:00.858781 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.858783    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:00.859024 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.858797    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:00.859024 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.858864    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:01.35884427 +0000 UTC m=+3.005383366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:00.861515 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.861490    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r85c\" (UniqueName: \"kubernetes.io/projected/740fa349-e5f2-4369-935e-8ee9ff32fb74-kube-api-access-5r85c\") pod \"node-ca-szttf\" (UID: \"740fa349-e5f2-4369-935e-8ee9ff32fb74\") " pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:00.861865 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.861848    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzfn\" (UniqueName: \"kubernetes.io/projected/a5395966-5e5b-4dca-a539-1281a2cc96df-kube-api-access-5vzfn\") pod \"multus-6fgxt\" (UID: \"a5395966-5e5b-4dca-a539-1281a2cc96df\") " pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:00.862389 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.862365    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjd5l\" (UniqueName: \"kubernetes.io/projected/48462f54-2be0-43f6-9c9d-d17214830d69-kube-api-access-cjd5l\") pod \"aws-ebs-csi-driver-node-bpn8n\" (UID: \"48462f54-2be0-43f6-9c9d-d17214830d69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:00.862621 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.862594    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljw98\" (UniqueName: \"kubernetes.io/projected/1aea3089-4245-4500-80c3-3fc0151b6c96-kube-api-access-ljw98\") pod \"multus-additional-cni-plugins-kzczk\" (UID: \"1aea3089-4245-4500-80c3-3fc0151b6c96\") " pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:00.863222 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.863195    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52846\" (UniqueName: \"kubernetes.io/projected/b6497f60-17b2-4e9c-bd83-17c0f13f3956-kube-api-access-52846\") pod \"tuned-qfn9v\" (UID: \"b6497f60-17b2-4e9c-bd83-17c0f13f3956\") " pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:00.863721 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.863698    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfd9\" (UniqueName: \"kubernetes.io/projected/42e7abb5-296e-4142-9ed6-0859b5613eb3-kube-api-access-fsfd9\") pod \"iptables-alerter-2q8zf\" (UID: \"42e7abb5-296e-4142-9ed6-0859b5613eb3\") " pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:00.864234 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.864191    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2bf\" (UniqueName: \"kubernetes.io/projected/70524e5b-7ba0-4696-b5c1-76288ad7c029-kube-api-access-wz2bf\") pod \"ovnkube-node-ftfml\" (UID: \"70524e5b-7ba0-4696-b5c1-76288ad7c029\") " pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:00.865001 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.864949    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqwg\" (UniqueName: \"kubernetes.io/projected/cd05c438-977e-42f1-ac3a-10605f8f2d41-kube-api-access-qwqwg\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:00.919371 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.919309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" event={"ID":"04dfa3a43b9c6e6928c1a69c4428406e","Type":"ContainerStarted","Data":"9bb84fb4cee2b76fed29ea3d38bc717f5b9df827bddcd80dc519d87c43e603d5"}
Jul 07 14:57:00.920517 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.920485    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"a84977041e1d635d44b56940b873b525","Type":"ContainerStarted","Data":"19d0459d3acf9536611d35a8c70d337e4df4ad6f5b088abefca8f4de90792b51"}
Jul 07 14:57:00.946844 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.946810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.946855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-dbus\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.946926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58e1c6b8-f561-4076-9157-299ed47c3827-agent-certs\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.946952    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.946967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58e1c6b8-f561-4076-9157-299ed47c3827-konnectivity-ca\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.946994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-kubelet-config\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.947026 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:00.947015    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:01.44699499 +0000 UTC m=+3.093534090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74cf5a19-c3f7-414c-963f-ac9a91e6f235-hosts-file\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947050    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-kubelet-config\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/74cf5a19-c3f7-414c-963f-ac9a91e6f235-kube-api-access-pgf8q\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cf5a19-c3f7-414c-963f-ac9a91e6f235-tmp-dir\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947091    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aafa307-bcb5-42de-953b-596ec07d8845-dbus\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:00.947316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947161    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/74cf5a19-c3f7-414c-963f-ac9a91e6f235-hosts-file\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.947586 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947345    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cf5a19-c3f7-414c-963f-ac9a91e6f235-tmp-dir\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:00.947775 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.947747    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58e1c6b8-f561-4076-9157-299ed47c3827-konnectivity-ca\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.949667 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.949649    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58e1c6b8-f561-4076-9157-299ed47c3827-agent-certs\") pod \"konnectivity-agent-zrkcb\" (UID: \"58e1c6b8-f561-4076-9157-299ed47c3827\") " pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:00.960127 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:00.960046    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgf8q\" (UniqueName: \"kubernetes.io/projected/74cf5a19-c3f7-414c-963f-ac9a91e6f235-kube-api-access-pgf8q\") pod \"node-resolver-2rvd7\" (UID: \"74cf5a19-c3f7-414c-963f-ac9a91e6f235\") " pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:01.029558 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.029519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n"
Jul 07 14:57:01.037856 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.037831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6fgxt"
Jul 07 14:57:01.048079 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.048027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kzczk"
Jul 07 14:57:01.052726 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.052702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qfn9v"
Jul 07 14:57:01.059660 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.059626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szttf"
Jul 07 14:57:01.068570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.068529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2q8zf"
Jul 07 14:57:01.076578 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.076557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:01.086494 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.086469    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:01.093355 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.093325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2rvd7"
Jul 07 14:57:01.292727 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.292640    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:01.351561 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.351522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:01.351721 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.351688    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:01.351782 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.351755    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:02.351732884 +0000 UTC m=+3.998271970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:01.452184 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.452160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:01.452333 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.452205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:01.452333 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452318    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:01.452333 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452330    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:01.452482 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452345    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:01.452482 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452356    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:01.452482 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452386    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:02.452366739 +0000 UTC m=+4.098905836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:01.452482 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:01.452402    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:02.452394686 +0000 UTC m=+4.098933771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:01.785959 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.785853    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:52:00 +0000 UTC" deadline="2028-03-24 17:40:51.758835072 +0000 UTC"
Jul 07 14:57:01.785959 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.785897    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15026h43m49.972942012s"
Jul 07 14:57:01.883300 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.883130    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cf5a19_c3f7_414c_963f_ac9a91e6f235.slice/crio-d31b95e1c7fb214e38c32faf2cc0ed88e45e844189ce3e3b98f7276fa21d59b9 WatchSource:0}: Error finding container d31b95e1c7fb214e38c32faf2cc0ed88e45e844189ce3e3b98f7276fa21d59b9: Status 404 returned error can't find the container with id d31b95e1c7fb214e38c32faf2cc0ed88e45e844189ce3e3b98f7276fa21d59b9
Jul 07 14:57:01.886263 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.886234    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5395966_5e5b_4dca_a539_1281a2cc96df.slice/crio-3d2b2c1eda69eb1c239280b75a70abf37c40069cba6a8030bc8d744c80337517 WatchSource:0}: Error finding container 3d2b2c1eda69eb1c239280b75a70abf37c40069cba6a8030bc8d744c80337517: Status 404 returned error can't find the container with id 3d2b2c1eda69eb1c239280b75a70abf37c40069cba6a8030bc8d744c80337517
Jul 07 14:57:01.886977 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.886953    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740fa349_e5f2_4369_935e_8ee9ff32fb74.slice/crio-937ca6843294370bd7d9e269e97e455b423c47616b60ae4e9072942050424f88 WatchSource:0}: Error finding container 937ca6843294370bd7d9e269e97e455b423c47616b60ae4e9072942050424f88: Status 404 returned error can't find the container with id 937ca6843294370bd7d9e269e97e455b423c47616b60ae4e9072942050424f88
Jul 07 14:57:01.887557 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.887529    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aea3089_4245_4500_80c3_3fc0151b6c96.slice/crio-b8a3367179fcf7028709270d7173d73e247b3f0cac9bd419b98b49c9172d6bb6 WatchSource:0}: Error finding container b8a3367179fcf7028709270d7173d73e247b3f0cac9bd419b98b49c9172d6bb6: Status 404 returned error can't find the container with id b8a3367179fcf7028709270d7173d73e247b3f0cac9bd419b98b49c9172d6bb6
Jul 07 14:57:01.888357 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.888242    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e7abb5_296e_4142_9ed6_0859b5613eb3.slice/crio-9347c8d59da3ef73a88964a8ca41e5084244404d8a4800b63dc5b7c3b36f72fd WatchSource:0}: Error finding container 9347c8d59da3ef73a88964a8ca41e5084244404d8a4800b63dc5b7c3b36f72fd: Status 404 returned error can't find the container with id 9347c8d59da3ef73a88964a8ca41e5084244404d8a4800b63dc5b7c3b36f72fd
Jul 07 14:57:01.890238 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.890030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e1c6b8_f561_4076_9157_299ed47c3827.slice/crio-e4dd7abfadfff9217f6d73db133b246c4546fd905736b2db7fba83cf53c4d130 WatchSource:0}: Error finding container e4dd7abfadfff9217f6d73db133b246c4546fd905736b2db7fba83cf53c4d130: Status 404 returned error can't find the container with id e4dd7abfadfff9217f6d73db133b246c4546fd905736b2db7fba83cf53c4d130
Jul 07 14:57:01.890921 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.890782    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48462f54_2be0_43f6_9c9d_d17214830d69.slice/crio-12955651ba133918c10023e7772df1b0eac571c69457b653009bba7411dfeb4b WatchSource:0}: Error finding container 12955651ba133918c10023e7772df1b0eac571c69457b653009bba7411dfeb4b: Status 404 returned error can't find the container with id 12955651ba133918c10023e7772df1b0eac571c69457b653009bba7411dfeb4b
Jul 07 14:57:01.892495 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:01.892281    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70524e5b_7ba0_4696_b5c1_76288ad7c029.slice/crio-83deda0e0d69febda636a4e5769a4d8bf8593279f05e6db6d5acdbd32a448b62 WatchSource:0}: Error finding container 83deda0e0d69febda636a4e5769a4d8bf8593279f05e6db6d5acdbd32a448b62: Status 404 returned error can't find the container with id 83deda0e0d69febda636a4e5769a4d8bf8593279f05e6db6d5acdbd32a448b62
Jul 07 14:57:01.926997 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.926967    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"83deda0e0d69febda636a4e5769a4d8bf8593279f05e6db6d5acdbd32a448b62"}
Jul 07 14:57:01.928122 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.928094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n" event={"ID":"48462f54-2be0-43f6-9c9d-d17214830d69","Type":"ContainerStarted","Data":"12955651ba133918c10023e7772df1b0eac571c69457b653009bba7411dfeb4b"}
Jul 07 14:57:01.929059 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.929037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrkcb" event={"ID":"58e1c6b8-f561-4076-9157-299ed47c3827","Type":"ContainerStarted","Data":"e4dd7abfadfff9217f6d73db133b246c4546fd905736b2db7fba83cf53c4d130"}
Jul 07 14:57:01.930053 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.930031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2q8zf" event={"ID":"42e7abb5-296e-4142-9ed6-0859b5613eb3","Type":"ContainerStarted","Data":"9347c8d59da3ef73a88964a8ca41e5084244404d8a4800b63dc5b7c3b36f72fd"}
Jul 07 14:57:01.930979 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.930959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6fgxt" event={"ID":"a5395966-5e5b-4dca-a539-1281a2cc96df","Type":"ContainerStarted","Data":"3d2b2c1eda69eb1c239280b75a70abf37c40069cba6a8030bc8d744c80337517"}
Jul 07 14:57:01.932342 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.932318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2rvd7" event={"ID":"74cf5a19-c3f7-414c-963f-ac9a91e6f235","Type":"ContainerStarted","Data":"d31b95e1c7fb214e38c32faf2cc0ed88e45e844189ce3e3b98f7276fa21d59b9"}
Jul 07 14:57:01.933333 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.933309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerStarted","Data":"b8a3367179fcf7028709270d7173d73e247b3f0cac9bd419b98b49c9172d6bb6"}
Jul 07 14:57:01.934326 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.934305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szttf" event={"ID":"740fa349-e5f2-4369-935e-8ee9ff32fb74","Type":"ContainerStarted","Data":"937ca6843294370bd7d9e269e97e455b423c47616b60ae4e9072942050424f88"}
Jul 07 14:57:01.935388 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:01.935368    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qfn9v" event={"ID":"b6497f60-17b2-4e9c-bd83-17c0f13f3956","Type":"ContainerStarted","Data":"6699667ae6cbec4155d687af8bf4049769efc05a42934ad87c8378f404b13df2"}
Jul 07 14:57:02.026863 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.026835    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:57:02.357149 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.357107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:02.357334 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.357242    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:02.357334 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.357310    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:04.35729372 +0000 UTC m=+6.003832813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:02.458009 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.457912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:02.458163 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.458015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:02.458163 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458148    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:02.458277 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458211    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:04.458192522 +0000 UTC m=+6.104731607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:02.458689 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458669    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:02.458771 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458693    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:02.458771 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458706    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:02.458771 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.458755    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:04.458739794 +0000 UTC m=+6.105278880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:02.786425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.786336    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:52:00 +0000 UTC" deadline="2028-01-21 10:36:57.049614938 +0000 UTC"
Jul 07 14:57:02.786425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.786378    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13507h39m54.263242594s"
Jul 07 14:57:02.926286 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.926253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:02.926737 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.926535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:02.927226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.926586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:02.934152 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.934119    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:02.934269 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.934213    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:02.934424 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:02.934404    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:02.950810 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.950771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" event={"ID":"04dfa3a43b9c6e6928c1a69c4428406e","Type":"ContainerStarted","Data":"1cc17f6308a21baa1fe33c0b35b6e3f2a467cb25d71c0b510e0ba94847104ce4"}
Jul 07 14:57:02.968177 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.968120    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" podStartSLOduration=2.968097919 podStartE2EDuration="2.968097919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:02.967227209 +0000 UTC m=+4.613766313" watchObservedRunningTime="2026-07-07 14:57:02.968097919 +0000 UTC m=+4.614637026"
Jul 07 14:57:02.970190 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.970161    2575 generic.go:356] "Generic (PLEG): container finished" podID="a84977041e1d635d44b56940b873b525" containerID="e1be3fd337096ad30a7ea8a19fa2443d2874b1bb8ce3dab07287639b42cb0af8" exitCode=0
Jul 07 14:57:02.970330 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:02.970210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"a84977041e1d635d44b56940b873b525","Type":"ContainerDied","Data":"e1be3fd337096ad30a7ea8a19fa2443d2874b1bb8ce3dab07287639b42cb0af8"}
Jul 07 14:57:03.998675 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:03.998640    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"a84977041e1d635d44b56940b873b525","Type":"ContainerStarted","Data":"e80d964cf92953e66c19c9a1225910a2990fb3640305174359aab4e4fe38e265"}
Jul 07 14:57:04.017688 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.017626    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" podStartSLOduration=4.017600881 podStartE2EDuration="4.017600881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:04.016457473 +0000 UTC m=+5.662996590" watchObservedRunningTime="2026-07-07 14:57:04.017600881 +0000 UTC m=+5.664140054"
Jul 07 14:57:04.375535 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.375330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:04.375535 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.375511    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:04.375756 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.375722    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:08.375665642 +0000 UTC m=+10.022204748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:04.476115 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.476076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:04.476278 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.476149    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:04.476353 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476303    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:04.476353 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476322    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:04.476353 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476335    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:04.476552 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476394    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:08.476374296 +0000 UTC m=+10.122913398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:04.476966 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476808    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:04.476966 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.476867    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:08.476851191 +0000 UTC m=+10.123390277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:04.916831 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.916802    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:04.917011 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.916923    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:04.917103 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.917066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:04.917219 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.917196    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:04.917276 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:04.917250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:04.917432 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:04.917348    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:06.914672 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:06.914580    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:06.915134 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:06.914734    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:06.915134 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:06.914599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:06.915134 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:06.914840    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:06.915134 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:06.914580    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:06.915134 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:06.914916    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:08.409047 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.409009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:08.409485 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.409163    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:08.409485 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.409235    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:16.409215568 +0000 UTC m=+18.055754652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:08.510426 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.510383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:08.510612 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.510462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:08.510612 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.510585    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:08.510723 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.510651    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:16.510631639 +0000 UTC m=+18.157170724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:08.511126 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.511101    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:08.511126 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.511134    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:08.511294 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.511150    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:08.511294 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.511208    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:16.511191211 +0000 UTC m=+18.157730308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.915092    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.915200    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.915556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.915652    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:08.915702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:08.915822 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:08.915781    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:10.914815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:10.914815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:10.914810    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:10.914995    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:10.915085    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:10.915336 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:10.915166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:12.914951 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:12.914839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:12.914951 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:12.914872    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:12.914951 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:12.914846    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:12.915501 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:12.914992    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:12.915501 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:12.915080    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:12.915501 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:12.915173    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:14.914549 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:14.914506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:14.914549 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:14.914551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:14.915110 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:14.914506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:14.915110 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:14.914640    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:14.915110 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:14.914727    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:14.915110 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:14.914804    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:16.469257 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.469174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:16.469783 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.469353    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:16.469783 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.469428    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:32.469410248 +0000 UTC m=+34.115949329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:16.569588 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.569546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:16.569776 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.569632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:16.569776 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569704    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:16.569890 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569787    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:32.569767856 +0000 UTC m=+34.216306948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:16.569890 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569787    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:16.569890 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569809    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:16.569890 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569820    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:16.569890 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.569856    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:57:32.569843906 +0000 UTC m=+34.216382990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:16.916126 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.916043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:16.916279 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.916048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:16.916279 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:16.916202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:16.916279 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.916211    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:16.916441 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.916287    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:16.916441 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:16.916390    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:18.915194 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:18.915145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:18.915657 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:18.915264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:18.915657 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:18.915270    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:18.915657 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:18.915401    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:18.915657 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:18.915470    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:18.915657 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:18.915554    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:20.915074 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:20.914831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:20.915411 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:20.915119    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:20.915411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:20.914991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:20.915411 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:20.915203    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:20.915411 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:20.914988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:20.915411 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:20.915289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:21.035404 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.035369    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"cd8e1bff7c3ac5636c8c6d7bc6a8f480a41640efb6423aca302af05a569f62d8"}
Jul 07 14:57:21.037101 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.037072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrkcb" event={"ID":"58e1c6b8-f561-4076-9157-299ed47c3827","Type":"ContainerStarted","Data":"0dca69c1e86c7a79cd5bc92afee6ad9d370638601afa8c311f7bf7856edfba5c"}
Jul 07 14:57:21.038830 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.038805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6fgxt" event={"ID":"a5395966-5e5b-4dca-a539-1281a2cc96df","Type":"ContainerStarted","Data":"d89a41c95b21a6073f1c1251e3ced40867dc18ac1e957f6b61f5047504fee628"}
Jul 07 14:57:21.040171 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.040134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2rvd7" event={"ID":"74cf5a19-c3f7-414c-963f-ac9a91e6f235","Type":"ContainerStarted","Data":"4d9bea32129a666cc3cc278f3f1388e6cea03392bd52f6d75ef5f9e921cc9d3b"}
Jul 07 14:57:21.041796 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.041767    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="dac003a661d1f710c491e0faddb35e8cf9ecd6d3328dcc2b670f7dfb658add70" exitCode=0
Jul 07 14:57:21.041901 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.041835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"dac003a661d1f710c491e0faddb35e8cf9ecd6d3328dcc2b670f7dfb658add70"}
Jul 07 14:57:21.045716 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.045572    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szttf" event={"ID":"740fa349-e5f2-4369-935e-8ee9ff32fb74","Type":"ContainerStarted","Data":"11aed5aa3de59dc999939244c247ae6dd0fc83331cdf83dbb89dc52b3b1dc7aa"}
Jul 07 14:57:21.047027 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.046992    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qfn9v" event={"ID":"b6497f60-17b2-4e9c-bd83-17c0f13f3956","Type":"ContainerStarted","Data":"1c433bf47205d8232422ec5b8b6b3d95513eb51f6f7824dc7e13a611aec65ccb"}
Jul 07 14:57:21.053342 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.053282    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zrkcb" podStartSLOduration=7.85327137 podStartE2EDuration="22.053239693s" totalImagesPullingTime="14.199968323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:21.053108669 +0000 UTC m=+22.699647772" watchObservedRunningTime="2026-07-07 14:57:21.053239693 +0000 UTC m=+22.699778797"
Jul 07 14:57:21.078121 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.078067    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2rvd7" podStartSLOduration=7.870619258 podStartE2EDuration="22.078046489s" totalImagesPullingTime="14.207427231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:21.077849464 +0000 UTC m=+22.724388567" watchObservedRunningTime="2026-07-07 14:57:21.078046489 +0000 UTC m=+22.724585595"
Jul 07 14:57:21.096907 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.096841    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6fgxt" podStartSLOduration=3.211419946 podStartE2EDuration="22.096821838s" totalImagesPullingTime="18.885401892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:21.096423229 +0000 UTC m=+22.742962334" watchObservedRunningTime="2026-07-07 14:57:21.096821838 +0000 UTC m=+22.743360942"
Jul 07 14:57:21.142754 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.142706    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qfn9v" podStartSLOduration=3.27801991 podStartE2EDuration="22.142689003s" totalImagesPullingTime="18.864669093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:21.142286911 +0000 UTC m=+22.788826015" watchObservedRunningTime="2026-07-07 14:57:21.142689003 +0000 UTC m=+22.789228105"
Jul 07 14:57:21.160576 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:21.160529    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-szttf" podStartSLOduration=3.305080113 podStartE2EDuration="22.160512968s" totalImagesPullingTime="18.855432855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:21.160199105 +0000 UTC m=+22.806738209" watchObservedRunningTime="2026-07-07 14:57:21.160512968 +0000 UTC m=+22.807052071"
Jul 07 14:57:22.055004 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.054955    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"5c600556a66830152ed4873f386decaadcb78c7fa6dcd487b02b9584d2411107"}
Jul 07 14:57:22.055893 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.055450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"39d53f31aab5e69b1ef8028a139b83b939f54e374f6f8629851bbdef39b5abbd"}
Jul 07 14:57:22.055893 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.055568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"09cbb88a787ea43cd7b0d4ca236f964c5b62ea92a27ed111a56c9a761cdb78e5"}
Jul 07 14:57:22.055893 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.055631    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"6b6754c0ed328edbf104522ecec2a41248b2a744812c03d187ef9b8ce855346b"}
Jul 07 14:57:22.055893 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.055685    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"fbacf85bef87a88e31e740916aa126141e7609d75e7e852c80315f4c4231f10a"}
Jul 07 14:57:22.060343 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.060306    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n" event={"ID":"48462f54-2be0-43f6-9c9d-d17214830d69","Type":"ContainerStarted","Data":"a326130a731b4c277b46ccd16bb6b6471f67c94de6c4c7a51712278fec060726"}
Jul 07 14:57:22.085996 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.085969    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 14:57:22.794124 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.793908    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:57:22.08599235Z","UUID":"3b60ac6c-5762-4b85-b1a9-161f9b112f48","Handler":null,"Name":"","Endpoint":""}
Jul 07 14:57:22.796299 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.795785    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 14:57:22.796299 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.795829    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 14:57:22.919427 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.919394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:22.919427 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.919418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:22.919725 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:22.919407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:22.919725 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:22.919542    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:22.919725 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:22.919621    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:22.919725 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:22.919718    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:23.065349 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:23.065320    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n" event={"ID":"48462f54-2be0-43f6-9c9d-d17214830d69","Type":"ContainerStarted","Data":"5bad77bb80cc4e8c2fa1270832aee15149f162542f5003a55f9d4c3b392e7caf"}
Jul 07 14:57:23.066964 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:23.066918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2q8zf" event={"ID":"42e7abb5-296e-4142-9ed6-0859b5613eb3","Type":"ContainerStarted","Data":"469a6518638784d9f99a294e1bed2e30998c007aa20c005520ea3e4a25a72b3a"}
Jul 07 14:57:23.108089 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:23.108030    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2q8zf" podStartSLOduration=5.254275882 podStartE2EDuration="24.108008028s" totalImagesPullingTime="18.853732146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:23.106374155 +0000 UTC m=+24.752913258" watchObservedRunningTime="2026-07-07 14:57:23.108008028 +0000 UTC m=+24.754547134"
Jul 07 14:57:23.897222 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:23.897179    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:23.897819 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:23.897794    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:24.072576 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.072537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"d8ece0d1415b36369c5bbed92c4a476600761c53f248c94d6a6228d4128abfbf"}
Jul 07 14:57:24.074851 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.074822    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n" event={"ID":"48462f54-2be0-43f6-9c9d-d17214830d69","Type":"ContainerStarted","Data":"92b1d064a9971188e41050b23705626f85d631c6be46c1a51fa1005be033bcd6"}
Jul 07 14:57:24.075261 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.075242    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:24.075643 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.075624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zrkcb"
Jul 07 14:57:24.125901 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.125836    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpn8n" podStartSLOduration=5.468312656 podStartE2EDuration="26.125812993s" totalImagesPullingTime="20.657500337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:24.104607826 +0000 UTC m=+25.751146931" watchObservedRunningTime="2026-07-07 14:57:24.125812993 +0000 UTC m=+25.772352097"
Jul 07 14:57:24.488417 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.488372    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2rvd7_74cf5a19-c3f7-414c-963f-ac9a91e6f235/dns-node-resolver/0.log"
Jul 07 14:57:24.915199 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.915108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:24.915199 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.915139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:24.915423 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:24.915251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:24.915634 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:24.915617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:24.915742 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:24.915721    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:24.915852 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:24.915822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:25.655617 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:25.655404    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szttf_740fa349-e5f2-4369-935e-8ee9ff32fb74/node-ca/0.log"
Jul 07 14:57:26.081807 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.081722    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="b4c31520c8c9075ed1c1c26e10206833dfbbc5ad95e58908601c671353b47015" exitCode=0
Jul 07 14:57:26.081966 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.081802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"b4c31520c8c9075ed1c1c26e10206833dfbbc5ad95e58908601c671353b47015"}
Jul 07 14:57:26.085312 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.085290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" event={"ID":"70524e5b-7ba0-4696-b5c1-76288ad7c029","Type":"ContainerStarted","Data":"f7496a5e2045e9c0b2465c472bfbc4688f54224548a27427b8a6461864b3c118"}
Jul 07 14:57:26.085616 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.085600    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:26.085676 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.085669    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:26.101823 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.101798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:26.171142 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.171090    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml" podStartSLOduration=8.259718510999999 podStartE2EDuration="27.171073834s" totalImagesPullingTime="18.911355323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:26.167636733 +0000 UTC m=+27.814175836" watchObservedRunningTime="2026-07-07 14:57:26.171073834 +0000 UTC m=+27.817612936"
Jul 07 14:57:26.915072 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.914715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:26.915825 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:26.915137    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:26.915825 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.915600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:26.915825 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:26.915709    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:26.916028 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:26.915838    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:26.916028 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:26.915919    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:27.088412 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.088384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:27.104361 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.104322    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:27.981051 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.981018    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gqtvk"]
Jul 07 14:57:27.981455 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.981143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:27.981455 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:27.981233    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:27.983372 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.983348    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt87x"]
Jul 07 14:57:27.983483 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.983449    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:27.983566 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:27.983545    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:27.996413 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.996377    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbkcv"]
Jul 07 14:57:27.996529 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:27.996478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:27.996581 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:27.996551    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:28.092754 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:28.092665    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="57990571c01cc00a71eef9f10a09a5cb037259f249ce9872918877f5a7968426" exitCode=0
Jul 07 14:57:28.092754 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:28.092736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"57990571c01cc00a71eef9f10a09a5cb037259f249ce9872918877f5a7968426"}
Jul 07 14:57:29.915268 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:29.914997    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:29.915664 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:29.915056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:29.915664 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:29.915289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:29.915664 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:29.915062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:29.915664 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:29.915420    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:29.915664 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:29.915512    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:30.102780 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:30.102746    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="6337a15a89f489c910f58902d8b3174b902450062d5056a7d888653b4bc85d03" exitCode=0
Jul 07 14:57:30.102925 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:30.102804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"6337a15a89f489c910f58902d8b3174b902450062d5056a7d888653b4bc85d03"}
Jul 07 14:57:31.914910 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:31.914881    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:31.915458 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:31.914884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:31.915458 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:31.915006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:31.915458 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:31.915006    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbkcv" podUID="1f878f43-2b0f-4463-968a-34052c7d9eef"
Jul 07 14:57:31.915458 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:31.915100    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt87x" podUID="2aafa307-bcb5-42de-953b-596ec07d8845"
Jul 07 14:57:31.915458 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:31.915194    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gqtvk" podUID="cd05c438-977e-42f1-ac3a-10605f8f2d41"
Jul 07 14:57:32.487301 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:32.487258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:32.487602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.487393    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:32.487602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.487465    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs podName:cd05c438-977e-42f1-ac3a-10605f8f2d41 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:04.487444591 +0000 UTC m=+66.133983675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs") pod "network-metrics-daemon-gqtvk" (UID: "cd05c438-977e-42f1-ac3a-10605f8f2d41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:57:32.588296 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:32.588257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:32.588488 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:32.588309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:32.588488 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588433    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:32.588602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588514    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret podName:2aafa307-bcb5-42de-953b-596ec07d8845 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:04.588490417 +0000 UTC m=+66.235029503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret") pod "global-pull-secret-syncer-lt87x" (UID: "2aafa307-bcb5-42de-953b-596ec07d8845") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:57:32.588602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588441    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:57:32.588602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588542    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:57:32.588602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588555    2575 projected.go:196] Error preparing data for projected volume kube-api-access-rdgv2 for pod openshift-network-diagnostics/network-check-target-rbkcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:32.588602 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:32.588604    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2 podName:1f878f43-2b0f-4463-968a-34052c7d9eef nodeName:}" failed. No retries permitted until 2026-07-07 14:58:04.58859003 +0000 UTC m=+66.235129117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdgv2" (UniqueName: "kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2") pod "network-check-target-rbkcv" (UID: "1f878f43-2b0f-4463-968a-34052c7d9eef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:57:33.611281 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.611196    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeReady"
Jul 07 14:57:33.611824 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.611365    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 14:57:33.667094 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.667061    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wc5dm"]
Jul 07 14:57:33.694044 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.694012    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6wjbk"]
Jul 07 14:57:33.694218 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.694138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.697752 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.697726    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pkm7x\""
Jul 07 14:57:33.698039 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.698022    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 14:57:33.698230 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.698216    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 14:57:33.709853 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.709829    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wc5dm"]
Jul 07 14:57:33.710004 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.709868    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wjbk"]
Jul 07 14:57:33.710004 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.709981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.712929 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.712900    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 14:57:33.713285 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.713262    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 14:57:33.713668 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.713649    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 14:57:33.714025 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.714007    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85x7v\""
Jul 07 14:57:33.714227 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.714210    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 14:57:33.770210 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.770172    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9hs6q"]
Jul 07 14:57:33.785128 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.785099    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:33.788181 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.788105    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 14:57:33.788387 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.788363    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hs6q"]
Jul 07 14:57:33.788570 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.788546    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 14:57:33.788760 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.788546    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bbd6b\""
Jul 07 14:57:33.788760 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.788590    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 14:57:33.797398 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.797530 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5h4\" (UniqueName: \"kubernetes.io/projected/ab6f2819-988c-484b-87db-2a7b9a7f5b58-kube-api-access-ss5h4\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.797530 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.797605 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797534    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117767b5-9775-4096-b6d2-ccc50508bcae-crio-socket\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.797605 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6f2819-988c-484b-87db-2a7b9a7f5b58-config-volume\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.797679 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117767b5-9775-4096-b6d2-ccc50508bcae-data-volume\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.797679 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfcl\" (UniqueName: \"kubernetes.io/projected/117767b5-9775-4096-b6d2-ccc50508bcae-kube-api-access-9hfcl\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.797679 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.797792 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.797708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6f2819-988c-484b-87db-2a7b9a7f5b58-tmp-dir\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.898431 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfcl\" (UniqueName: \"kubernetes.io/projected/117767b5-9775-4096-b6d2-ccc50508bcae-kube-api-access-9hfcl\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.898431 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.898431 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6f2819-988c-484b-87db-2a7b9a7f5b58-tmp-dir\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0720eaae-7119-4509-93a1-5660aa611213-cert\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898577    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5h4\" (UniqueName: \"kubernetes.io/projected/ab6f2819-988c-484b-87db-2a7b9a7f5b58-kube-api-access-ss5h4\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898637    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls podName:ab6f2819-988c-484b-87db-2a7b9a7f5b58 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:34.398617841 +0000 UTC m=+36.045156929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls") pod "dns-default-wc5dm" (UID: "ab6f2819-988c-484b-87db-2a7b9a7f5b58") : secret "dns-default-metrics-tls" not found
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd4vl\" (UniqueName: \"kubernetes.io/projected/0720eaae-7119-4509-93a1-5660aa611213-kube-api-access-cd4vl\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898681    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:33.898701 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117767b5-9775-4096-b6d2-ccc50508bcae-crio-socket\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898723    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:34.398713008 +0000 UTC m=+36.045252107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab6f2819-988c-484b-87db-2a7b9a7f5b58-tmp-dir\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6f2819-988c-484b-87db-2a7b9a7f5b58-config-volume\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117767b5-9775-4096-b6d2-ccc50508bcae-data-volume\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898577    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:33.898888    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:34.398873655 +0000 UTC m=+36.045412737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : configmap "kube-rbac-proxy" not found
Jul 07 14:57:33.899192 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.898923    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117767b5-9775-4096-b6d2-ccc50508bcae-crio-socket\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.899425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.899196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117767b5-9775-4096-b6d2-ccc50508bcae-data-volume\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.899425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.899258    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6f2819-988c-484b-87db-2a7b9a7f5b58-config-volume\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.913695 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.913666    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfcl\" (UniqueName: \"kubernetes.io/projected/117767b5-9775-4096-b6d2-ccc50508bcae-kube-api-access-9hfcl\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:33.913862 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.913676    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5h4\" (UniqueName: \"kubernetes.io/projected/ab6f2819-988c-484b-87db-2a7b9a7f5b58-kube-api-access-ss5h4\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:33.914470 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.914450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:57:33.914578 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.914557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:57:33.914845 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.914833    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:57:33.917286 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.917263    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:57:33.917998 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.917800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fcz7\""
Jul 07 14:57:33.918487 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.918341    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:57:33.918583 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.918489    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:57:33.918583 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.918507    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jrv9k\""
Jul 07 14:57:33.918583 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.918521    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:57:33.994286 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.994249    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-l2brz"]
Jul 07 14:57:33.999410 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.999381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0720eaae-7119-4509-93a1-5660aa611213-cert\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:33.999578 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:33.999472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd4vl\" (UniqueName: \"kubernetes.io/projected/0720eaae-7119-4509-93a1-5660aa611213-kube-api-access-cd4vl\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:34.002426 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.002401    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0720eaae-7119-4509-93a1-5660aa611213-cert\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:34.013923 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.013898    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd4vl\" (UniqueName: \"kubernetes.io/projected/0720eaae-7119-4509-93a1-5660aa611213-kube-api-access-cd4vl\") pod \"ingress-canary-9hs6q\" (UID: \"0720eaae-7119-4509-93a1-5660aa611213\") " pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:34.027213 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.027183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-l2brz"]
Jul 07 14:57:34.027383 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.027320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.037457 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.037428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 14:57:34.037613 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.037505    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 14:57:34.037613 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.037513    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 14:57:34.037867 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.037850    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9ldns\""
Jul 07 14:57:34.038056 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.038035    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 14:57:34.096767 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.096724    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hs6q"
Jul 07 14:57:34.100415 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.100356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-cabundle\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.103304 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.100687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzb5\" (UniqueName: \"kubernetes.io/projected/c6a8c286-e187-440d-a210-1f499e5dcd0c-kube-api-access-fqzb5\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.103304 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.100731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-key\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.202179 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.202140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzb5\" (UniqueName: \"kubernetes.io/projected/c6a8c286-e187-440d-a210-1f499e5dcd0c-kube-api-access-fqzb5\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.202316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.202194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-key\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.202316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.202237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-cabundle\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.203064 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.203034    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-cabundle\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.204900 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.204872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c6a8c286-e187-440d-a210-1f499e5dcd0c-signing-key\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.222874 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.222841    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzb5\" (UniqueName: \"kubernetes.io/projected/c6a8c286-e187-440d-a210-1f499e5dcd0c-kube-api-access-fqzb5\") pod \"service-ca-758f599c5-l2brz\" (UID: \"c6a8c286-e187-440d-a210-1f499e5dcd0c\") " pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.337525 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.337480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-l2brz"
Jul 07 14:57:34.391906 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.391871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hs6q"]
Jul 07 14:57:34.405074 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.405047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:34.405256 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.405088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:34.405256 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.405166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:34.405256 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405203    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:57:34.405256 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405239    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:57:34.405446 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405260    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls podName:ab6f2819-988c-484b-87db-2a7b9a7f5b58 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:35.405242717 +0000 UTC m=+37.051781809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls") pod "dns-default-wc5dm" (UID: "ab6f2819-988c-484b-87db-2a7b9a7f5b58") : secret "dns-default-metrics-tls" not found
Jul 07 14:57:34.405446 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405276    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:35.405266473 +0000 UTC m=+37.051805570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : configmap "kube-rbac-proxy" not found
Jul 07 14:57:34.405446 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405337    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:34.405446 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:34.405411    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:35.405390561 +0000 UTC m=+37.051929644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:34.439378 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:34.439342    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0720eaae_7119_4509_93a1_5660aa611213.slice/crio-3b909bf023c3bff09479593db87205e9e2a516f90f8aa31c52f7898ebf5bf90e WatchSource:0}: Error finding container 3b909bf023c3bff09479593db87205e9e2a516f90f8aa31c52f7898ebf5bf90e: Status 404 returned error can't find the container with id 3b909bf023c3bff09479593db87205e9e2a516f90f8aa31c52f7898ebf5bf90e
Jul 07 14:57:34.577924 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:34.577897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-l2brz"]
Jul 07 14:57:34.581559 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:34.581525    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a8c286_e187_440d_a210_1f499e5dcd0c.slice/crio-0e1fb17ba671391155eda6e753601a55aaac6913b8b6bfca15c9cb34ec760914 WatchSource:0}: Error finding container 0e1fb17ba671391155eda6e753601a55aaac6913b8b6bfca15c9cb34ec760914: Status 404 returned error can't find the container with id 0e1fb17ba671391155eda6e753601a55aaac6913b8b6bfca15c9cb34ec760914
Jul 07 14:57:35.118850 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.118819    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="11575d2aab10c3598ab99cd8f27debe64b4e5f0bc3468918e7939601aef6199f" exitCode=0
Jul 07 14:57:35.119456 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.118900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"11575d2aab10c3598ab99cd8f27debe64b4e5f0bc3468918e7939601aef6199f"}
Jul 07 14:57:35.120590 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.120556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hs6q" event={"ID":"0720eaae-7119-4509-93a1-5660aa611213","Type":"ContainerStarted","Data":"3b909bf023c3bff09479593db87205e9e2a516f90f8aa31c52f7898ebf5bf90e"}
Jul 07 14:57:35.121802 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.121776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-l2brz" event={"ID":"c6a8c286-e187-440d-a210-1f499e5dcd0c","Type":"ContainerStarted","Data":"0e1fb17ba671391155eda6e753601a55aaac6913b8b6bfca15c9cb34ec760914"}
Jul 07 14:57:35.413224 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.413122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:35.413224 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.413190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:35.413414 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:35.413248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:35.413414 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413356    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:57:35.413502 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413418    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:37.41340014 +0000 UTC m=+39.059939226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : configmap "kube-rbac-proxy" not found
Jul 07 14:57:35.413689 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413662    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:57:35.413825 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413739    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls podName:ab6f2819-988c-484b-87db-2a7b9a7f5b58 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:37.413711623 +0000 UTC m=+39.060250704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls") pod "dns-default-wc5dm" (UID: "ab6f2819-988c-484b-87db-2a7b9a7f5b58") : secret "dns-default-metrics-tls" not found
Jul 07 14:57:35.413825 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413662    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:35.413825 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:35.413799    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:37.413788038 +0000 UTC m=+39.060327140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:36.129677 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:36.128868    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aea3089-4245-4500-80c3-3fc0151b6c96" containerID="c454ea4bfa92b4ec9e193334a7040a96d274fcd0c49737cf19895de146511d82" exitCode=0
Jul 07 14:57:36.129677 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:36.128969    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerDied","Data":"c454ea4bfa92b4ec9e193334a7040a96d274fcd0c49737cf19895de146511d82"}
Jul 07 14:57:37.432384 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:37.432145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:37.432418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:37.432293    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:37.432478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:37.432516    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:41.432496988 +0000 UTC m=+43.079036069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:37.432602    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:57:37.432871 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:37.432665    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls podName:ab6f2819-988c-484b-87db-2a7b9a7f5b58 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:41.432646515 +0000 UTC m=+43.079185598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls") pod "dns-default-wc5dm" (UID: "ab6f2819-988c-484b-87db-2a7b9a7f5b58") : secret "dns-default-metrics-tls" not found
Jul 07 14:57:37.433216 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:37.433197    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117767b5-9775-4096-b6d2-ccc50508bcae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:38.135540 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.135498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-l2brz" event={"ID":"c6a8c286-e187-440d-a210-1f499e5dcd0c","Type":"ContainerStarted","Data":"fbab031a87709aff52283fbe6bd31a716453255d23367d614f5b5e225f72acfc"}
Jul 07 14:57:38.138963 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.138915    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzczk" event={"ID":"1aea3089-4245-4500-80c3-3fc0151b6c96","Type":"ContainerStarted","Data":"addd1de23827128efd02d371a97d98de05569693bd5e8270763da0c294be6c52"}
Jul 07 14:57:38.140400 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.140367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hs6q" event={"ID":"0720eaae-7119-4509-93a1-5660aa611213","Type":"ContainerStarted","Data":"3909b9cc57fffb21b0276b03269180a394d7003af2e1d6bcdbf23854f5005cf7"}
Jul 07 14:57:38.155183 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.155131    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-l2brz" podStartSLOduration=2.479413709 podStartE2EDuration="5.155110011s" totalImagesPullingTime="2.675696302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:38.154082959 +0000 UTC m=+39.800622063" watchObservedRunningTime="2026-07-07 14:57:38.155110011 +0000 UTC m=+39.801649115"
Jul 07 14:57:38.171480 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.171422    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9hs6q" podStartSLOduration=2.360762433 podStartE2EDuration="5.171400276s" totalImagesPullingTime="2.810637843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:38.170399271 +0000 UTC m=+39.816938389" watchObservedRunningTime="2026-07-07 14:57:38.171400276 +0000 UTC m=+39.817939375"
Jul 07 14:57:38.193836 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:38.193774    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kzczk" podStartSLOduration=9.570587932 podStartE2EDuration="39.19375183s" totalImagesPullingTime="29.623163898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:38.193131965 +0000 UTC m=+39.839671068" watchObservedRunningTime="2026-07-07 14:57:38.19375183 +0000 UTC m=+39.840290934"
Jul 07 14:57:41.462320 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:41.462273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:41.462812 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:41.462348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:41.462812 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:41.462427    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:41.462812 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:41.462501    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls podName:117767b5-9775-4096-b6d2-ccc50508bcae nodeName:}" failed. No retries permitted until 2026-07-07 14:57:49.462483066 +0000 UTC m=+51.109022151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6wjbk" (UID: "117767b5-9775-4096-b6d2-ccc50508bcae") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:57:41.462812 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:41.462518    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:57:41.462812 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:41.462581    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls podName:ab6f2819-988c-484b-87db-2a7b9a7f5b58 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:49.462563997 +0000 UTC m=+51.109103098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls") pod "dns-default-wc5dm" (UID: "ab6f2819-988c-484b-87db-2a7b9a7f5b58") : secret "dns-default-metrics-tls" not found
Jul 07 14:57:49.531246 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.531204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:49.531704 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.531257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:49.535380 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.535342    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab6f2819-988c-484b-87db-2a7b9a7f5b58-metrics-tls\") pod \"dns-default-wc5dm\" (UID: \"ab6f2819-988c-484b-87db-2a7b9a7f5b58\") " pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:49.535511 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.535342    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117767b5-9775-4096-b6d2-ccc50508bcae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6wjbk\" (UID: \"117767b5-9775-4096-b6d2-ccc50508bcae\") " pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:49.604587 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.604544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:49.621735 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.621701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6wjbk"
Jul 07 14:57:49.782033 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.781904    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wc5dm"]
Jul 07 14:57:49.786091 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:49.786065    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6f2819_988c_484b_87db_2a7b9a7f5b58.slice/crio-f0f327db6c1a344a3cc3619838dd54711eab94ae9aacb347ef9bad30c0d701ed WatchSource:0}: Error finding container f0f327db6c1a344a3cc3619838dd54711eab94ae9aacb347ef9bad30c0d701ed: Status 404 returned error can't find the container with id f0f327db6c1a344a3cc3619838dd54711eab94ae9aacb347ef9bad30c0d701ed
Jul 07 14:57:49.797766 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:49.797742    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6wjbk"]
Jul 07 14:57:49.800824 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:49.800799    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117767b5_9775_4096_b6d2_ccc50508bcae.slice/crio-ebe698c32dedbe2e4b57439977c5bcc5f2b9e09d5d9742bddc256aee874c421e WatchSource:0}: Error finding container ebe698c32dedbe2e4b57439977c5bcc5f2b9e09d5d9742bddc256aee874c421e: Status 404 returned error can't find the container with id ebe698c32dedbe2e4b57439977c5bcc5f2b9e09d5d9742bddc256aee874c421e
Jul 07 14:57:50.173387 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:50.173299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wjbk" event={"ID":"117767b5-9775-4096-b6d2-ccc50508bcae","Type":"ContainerStarted","Data":"ebe698c32dedbe2e4b57439977c5bcc5f2b9e09d5d9742bddc256aee874c421e"}
Jul 07 14:57:50.174455 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:50.174427    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc5dm" event={"ID":"ab6f2819-988c-484b-87db-2a7b9a7f5b58","Type":"ContainerStarted","Data":"f0f327db6c1a344a3cc3619838dd54711eab94ae9aacb347ef9bad30c0d701ed"}
Jul 07 14:57:53.716493 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.716458    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"]
Jul 07 14:57:53.738916 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.738874    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"]
Jul 07 14:57:53.739110 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.739015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.741599 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.741573    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 14:57:53.742249 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742225    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 14:57:53.742249 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742242    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 14:57:53.742562 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742470    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 14:57:53.743007 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 14:57:53.743104 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742980    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 14:57:53.743104 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.742986    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 14:57:53.761111 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.761218 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.761218 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.761218 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndfp\" (UniqueName: \"kubernetes.io/projected/5bdeca56-0ce4-44a4-8dba-32fece18d588-kube-api-access-jndfp\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.761317 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.761317 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.761247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdeca56-0ce4-44a4-8dba-32fece18d588-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.861656 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.861656 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.861874 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.861874 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndfp\" (UniqueName: \"kubernetes.io/projected/5bdeca56-0ce4-44a4-8dba-32fece18d588-kube-api-access-jndfp\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.861874 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.862047 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.861881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdeca56-0ce4-44a4-8dba-32fece18d588-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.862618 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.862595    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdeca56-0ce4-44a4-8dba-32fece18d588-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.864223 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.864196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.864385 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.864366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.864580 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.864564    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-ca\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.864744 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.864729    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bdeca56-0ce4-44a4-8dba-32fece18d588-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.883628 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.883595    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndfp\" (UniqueName: \"kubernetes.io/projected/5bdeca56-0ce4-44a4-8dba-32fece18d588-kube-api-access-jndfp\") pod \"cluster-proxy-proxy-agent-5b59844d64-nm698\" (UID: \"5bdeca56-0ce4-44a4-8dba-32fece18d588\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:53.929183 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.929153    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc7df4966-ljnd4"]
Jul 07 14:57:53.960657 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.960625    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc7df4966-ljnd4"]
Jul 07 14:57:53.960807 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.960757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963085 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55a9ebce-7148-4feb-93a7-cb0158b5754e-ca-trust-extracted\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963085 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963086    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 14:57:53.963085 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963094    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mrpzk\""
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-bound-sa-token\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-trusted-ca\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-tls\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpsk\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-kube-api-access-pqpsk\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-certificates\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-installation-pull-secrets\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-image-registry-private-configuration\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:53.963531 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963379    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 14:57:53.963802 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.963779    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 14:57:53.968324 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:53.968268    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 14:57:54.063880 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.063846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-tls\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064090 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.063891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpsk\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-kube-api-access-pqpsk\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064090 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.063912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-certificates\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064275 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-installation-pull-secrets\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064435 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-image-registry-private-configuration\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064435 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55a9ebce-7148-4feb-93a7-cb0158b5754e-ca-trust-extracted\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064435 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-bound-sa-token\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.064605 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-trusted-ca\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.065018 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064676    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-certificates\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.065018 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.064831    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55a9ebce-7148-4feb-93a7-cb0158b5754e-ca-trust-extracted\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.065339 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.065297    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55a9ebce-7148-4feb-93a7-cb0158b5754e-trusted-ca\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.066726 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.066333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"
Jul 07 14:57:54.067239 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.067213    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-registry-tls\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.067375 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.067354    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-image-registry-private-configuration\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.067459 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.067436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55a9ebce-7148-4feb-93a7-cb0158b5754e-installation-pull-secrets\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.074094 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.074063    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpsk\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-kube-api-access-pqpsk\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.079154 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.079127    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55a9ebce-7148-4feb-93a7-cb0158b5754e-bound-sa-token\") pod \"image-registry-5cc7df4966-ljnd4\" (UID: \"55a9ebce-7148-4feb-93a7-cb0158b5754e\") " pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:54.271110 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:54.271079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:55.154722 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:55.154693    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698"]
Jul 07 14:57:55.157188 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:55.156826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc7df4966-ljnd4"]
Jul 07 14:57:55.159488 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:55.159446    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bdeca56_0ce4_44a4_8dba_32fece18d588.slice/crio-c120354cc5dfebf8f76f067cea1171aab58aa5f3a353dd90159ebab5bb95572d WatchSource:0}: Error finding container c120354cc5dfebf8f76f067cea1171aab58aa5f3a353dd90159ebab5bb95572d: Status 404 returned error can't find the container with id c120354cc5dfebf8f76f067cea1171aab58aa5f3a353dd90159ebab5bb95572d
Jul 07 14:57:55.161726 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:55.160869    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a9ebce_7148_4feb_93a7_cb0158b5754e.slice/crio-bfca40a6c79e7d4ea13626ccb7f96ceb8a9c10df66be295608a5ca1ace376cea WatchSource:0}: Error finding container bfca40a6c79e7d4ea13626ccb7f96ceb8a9c10df66be295608a5ca1ace376cea: Status 404 returned error can't find the container with id bfca40a6c79e7d4ea13626ccb7f96ceb8a9c10df66be295608a5ca1ace376cea
Jul 07 14:57:55.196585 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:55.196542    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698" event={"ID":"5bdeca56-0ce4-44a4-8dba-32fece18d588","Type":"ContainerStarted","Data":"c120354cc5dfebf8f76f067cea1171aab58aa5f3a353dd90159ebab5bb95572d"}
Jul 07 14:57:55.197844 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:55.197818    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc5dm" event={"ID":"ab6f2819-988c-484b-87db-2a7b9a7f5b58","Type":"ContainerStarted","Data":"149022d11364fdd01502e2fdbda4641ffeadcd87de7d0124fc3a66425dec8367"}
Jul 07 14:57:55.198926 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:55.198904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4" event={"ID":"55a9ebce-7148-4feb-93a7-cb0158b5754e","Type":"ContainerStarted","Data":"bfca40a6c79e7d4ea13626ccb7f96ceb8a9c10df66be295608a5ca1ace376cea"}
Jul 07 14:57:56.205595 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.205557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wjbk" event={"ID":"117767b5-9775-4096-b6d2-ccc50508bcae","Type":"ContainerStarted","Data":"62717fb50f37fededede621e672a339035fe9b38fe537af6bfdb6f679b461661"}
Jul 07 14:57:56.207922 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.207880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wc5dm" event={"ID":"ab6f2819-988c-484b-87db-2a7b9a7f5b58","Type":"ContainerStarted","Data":"f29a9f9ec9be16d293049a4da437ebfcc76e8ee8bf92fc553f68d7126c085b33"}
Jul 07 14:57:56.209539 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.209511    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4" event={"ID":"55a9ebce-7148-4feb-93a7-cb0158b5754e","Type":"ContainerStarted","Data":"58cad4d2d78a86e29ce0ff44a0310b8d6ef97d9dcaf5f32755ba2dba30c1f186"}
Jul 07 14:57:56.209669 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.209609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:57:56.228898 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.228145    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wc5dm" podStartSLOduration=18.029385707 podStartE2EDuration="23.228126659s" totalImagesPullingTime="5.198740952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:56.22684346 +0000 UTC m=+57.873382558" watchObservedRunningTime="2026-07-07 14:57:56.228126659 +0000 UTC m=+57.874665810"
Jul 07 14:57:56.250278 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:56.250212    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4" podStartSLOduration=3.250189035 podStartE2EDuration="3.250189035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:56.248961365 +0000 UTC m=+57.895500469" watchObservedRunningTime="2026-07-07 14:57:56.250189035 +0000 UTC m=+57.896728139"
Jul 07 14:57:57.216319 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:57.216195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wjbk" event={"ID":"117767b5-9775-4096-b6d2-ccc50508bcae","Type":"ContainerStarted","Data":"a3ffb6dc54f2bb898d1b9a41f3f7324c41ef0812873129f1d615d20646fda63b"}
Jul 07 14:57:57.216779 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:57.216740    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:57:58.020765 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.020718    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"]
Jul 07 14:57:58.034337 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.034309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.036472 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.036440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"]
Jul 07 14:57:58.036908 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.036883    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tcvtg\""
Jul 07 14:57:58.037127 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.037107    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 14:57:58.037201 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.037179    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 14:57:58.037968 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.037918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 14:57:58.038069 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.038046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 14:57:58.038385 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.038237    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 14:57:58.098898 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.098860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257t2\" (UniqueName: \"kubernetes.io/projected/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-kube-api-access-257t2\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.098898 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.098908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.099207 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.099044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.099531 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.099496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.200634 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.200359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.200634 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.200432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257t2\" (UniqueName: \"kubernetes.io/projected/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-kube-api-access-257t2\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.200634 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.200461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.200634 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.200524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.200972 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:58.200671    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 14:57:58.200972 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:57:58.200749    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls podName:60a05fda-c0dc-495a-8c3c-f4f8e836c43b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:58.70072933 +0000 UTC m=+60.347268426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-2fm26" (UID: "60a05fda-c0dc-495a-8c3c-f4f8e836c43b") : secret "prometheus-operator-tls" not found
Jul 07 14:57:58.201331 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.201310    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.203359 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.203329    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.210097 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.210073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257t2\" (UniqueName: \"kubernetes.io/projected/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-kube-api-access-257t2\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.703948 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.703888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.706844 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.706820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/60a05fda-c0dc-495a-8c3c-f4f8e836c43b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2fm26\" (UID: \"60a05fda-c0dc-495a-8c3c-f4f8e836c43b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:58.949285 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.949250    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tcvtg\""
Jul 07 14:57:58.957497 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:58.957421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"
Jul 07 14:57:59.132082 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:59.132051    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ftfml"
Jul 07 14:57:59.237552 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:57:59.237528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26"]
Jul 07 14:57:59.240003 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:57:59.239969    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a05fda_c0dc_495a_8c3c_f4f8e836c43b.slice/crio-963f629f8c786ae89fe15c3938740613e1e563ce56cfe09c5b3c6b3bcd46297b WatchSource:0}: Error finding container 963f629f8c786ae89fe15c3938740613e1e563ce56cfe09c5b3c6b3bcd46297b: Status 404 returned error can't find the container with id 963f629f8c786ae89fe15c3938740613e1e563ce56cfe09c5b3c6b3bcd46297b
Jul 07 14:58:00.233827 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:00.233737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26" event={"ID":"60a05fda-c0dc-495a-8c3c-f4f8e836c43b","Type":"ContainerStarted","Data":"963f629f8c786ae89fe15c3938740613e1e563ce56cfe09c5b3c6b3bcd46297b"}
Jul 07 14:58:00.236366 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:00.236338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698" event={"ID":"5bdeca56-0ce4-44a4-8dba-32fece18d588","Type":"ContainerStarted","Data":"7c510d55172258f099bbe87d280d9ea493f2f5361f633a0b77e24683434574c8"}
Jul 07 14:58:00.236501 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:00.236381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698" event={"ID":"5bdeca56-0ce4-44a4-8dba-32fece18d588","Type":"ContainerStarted","Data":"03639969606ad3adf21970b753f5477a5daa4732d0aea9de933de40cca5caf1c"}
Jul 07 14:58:00.236501 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:00.236399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698" event={"ID":"5bdeca56-0ce4-44a4-8dba-32fece18d588","Type":"ContainerStarted","Data":"96459b4ec2ad3cf126208978ba614dd4218da4b77f304a76f1091fe966ad4dff"}
Jul 07 14:58:00.256590 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:00.256416    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b59844d64-nm698" podStartSLOduration=2.947191938 podStartE2EDuration="7.256398055s" totalImagesPullingTime="4.309206117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:00.255327902 +0000 UTC m=+61.901867006" watchObservedRunningTime="2026-07-07 14:58:00.256398055 +0000 UTC m=+61.902937152"
Jul 07 14:58:02.244706 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:02.244616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26" event={"ID":"60a05fda-c0dc-495a-8c3c-f4f8e836c43b","Type":"ContainerStarted","Data":"d37bb2324b07614e2386ce0992845be02c14ded9a8a278d93b1a5f8bee3d329a"}
Jul 07 14:58:02.244706 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:02.244675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26" event={"ID":"60a05fda-c0dc-495a-8c3c-f4f8e836c43b","Type":"ContainerStarted","Data":"732c5ba9c3d19494a6f3a5a7b88601c7ec8b6c0103b8afbbf4dd47d44ad86761"}
Jul 07 14:58:02.246804 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:02.246780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6wjbk" event={"ID":"117767b5-9775-4096-b6d2-ccc50508bcae","Type":"ContainerStarted","Data":"532e002c3f0e5f3a22a06c53e2dc2dcd842fb2c7bf24d607a41889953297b0ed"}
Jul 07 14:58:02.279092 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:02.279038    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2fm26" podStartSLOduration=2.7049577019999997 podStartE2EDuration="5.27902207s" totalImagesPullingTime="2.574064368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:02.278580329 +0000 UTC m=+63.925119432" watchObservedRunningTime="2026-07-07 14:58:02.27902207 +0000 UTC m=+63.925561173"
Jul 07 14:58:02.316340 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:02.316283    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6wjbk" podStartSLOduration=17.520652439 podStartE2EDuration="29.316269045s" totalImagesPullingTime="11.795616606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:02.316188711 +0000 UTC m=+63.962727815" watchObservedRunningTime="2026-07-07 14:58:02.316269045 +0000 UTC m=+63.962808147"
Jul 07 14:58:04.442813 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.442778    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-597sm"]
Jul 07 14:58:04.445300 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.445280    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.447575 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.447540    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ggm5j\""
Jul 07 14:58:04.447894 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.447870    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 14:58:04.448524 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.448500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 14:58:04.448616 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.448512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 14:58:04.562963 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.562896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-textfile\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.562963 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.562960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.562987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-metrics-client-ca\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-root\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:58:04.563226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-sys\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-wtmp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563295    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.563399 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.563383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jhp\" (UniqueName: \"kubernetes.io/projected/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-kube-api-access-z8jhp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.565909 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.565891    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:58:04.576418 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.576382    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd05c438-977e-42f1-ac3a-10605f8f2d41-metrics-certs\") pod \"network-metrics-daemon-gqtvk\" (UID: \"cd05c438-977e-42f1-ac3a-10605f8f2d41\") " pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:58:04.664425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-wtmp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664425 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jhp\" (UniqueName: \"kubernetes.io/projected/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-kube-api-access-z8jhp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-textfile\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.664607    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-wtmp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:58:04.664660    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 14:58:04.664803 ip-10-0-128-215 kubenswrapper[2575]: E0707 14:58:04.664732    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls podName:aeab4723-c9f2-42e8-b3bf-fe0dea37a35b nodeName:}" failed. No retries permitted until 2026-07-07 14:58:05.164711019 +0000 UTC m=+66.811250107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls") pod "node-exporter-597sm" (UID: "aeab4723-c9f2-42e8-b3bf-fe0dea37a35b") : secret "node-exporter-tls" not found
Jul 07 14:58:04.665233 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-accelerators-collector-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665233 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-textfile\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665233 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-metrics-client-ca\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665377 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-root\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665377 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:58:04.665377 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:58:04.665516 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-sys\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665516 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-root\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665516 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-sys\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.665735 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.665715    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-metrics-client-ca\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.667659 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.667635    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:58:04.668595 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.668030    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:58:04.668595 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.668449    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.677793 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.677766    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jhp\" (UniqueName: \"kubernetes.io/projected/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-kube-api-access-z8jhp\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:04.678308 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.678005    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:58:04.678418 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.678246    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aafa307-bcb5-42de-953b-596ec07d8845-original-pull-secret\") pod \"global-pull-secret-syncer-lt87x\" (UID: \"2aafa307-bcb5-42de-953b-596ec07d8845\") " pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:58:04.688778 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.688754    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdgv2\" (UniqueName: \"kubernetes.io/projected/1f878f43-2b0f-4463-968a-34052c7d9eef-kube-api-access-rdgv2\") pod \"network-check-target-rbkcv\" (UID: \"1f878f43-2b0f-4463-968a-34052c7d9eef\") " pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:58:04.833820 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.833782    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8fcz7\""
Jul 07 14:58:04.841294 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.841267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gqtvk"
Jul 07 14:58:04.842737 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.842720    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jrv9k\""
Jul 07 14:58:04.846316 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.846292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt87x"
Jul 07 14:58:04.850286 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:04.850269    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:58:05.003277 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.003190    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt87x"]
Jul 07 14:58:05.008427 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:58:05.008389    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aafa307_bcb5_42de_953b_596ec07d8845.slice/crio-2bc4df752b7a0a28c4fc152335a79a38a153ae1505f3a36b8eb29b65566a4616 WatchSource:0}: Error finding container 2bc4df752b7a0a28c4fc152335a79a38a153ae1505f3a36b8eb29b65566a4616: Status 404 returned error can't find the container with id 2bc4df752b7a0a28c4fc152335a79a38a153ae1505f3a36b8eb29b65566a4616
Jul 07 14:58:05.031217 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.031182    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gqtvk"]
Jul 07 14:58:05.034006 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.033980    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbkcv"]
Jul 07 14:58:05.034321 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:58:05.034299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd05c438_977e_42f1_ac3a_10605f8f2d41.slice/crio-4fdd2d5ead322c4da69d58caac2ceb573a3670c9d668c039dd1a46fd9449a558 WatchSource:0}: Error finding container 4fdd2d5ead322c4da69d58caac2ceb573a3670c9d668c039dd1a46fd9449a558: Status 404 returned error can't find the container with id 4fdd2d5ead322c4da69d58caac2ceb573a3670c9d668c039dd1a46fd9449a558
Jul 07 14:58:05.037328 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:58:05.037299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f878f43_2b0f_4463_968a_34052c7d9eef.slice/crio-ec7426e203d77709c23bfbabf8d421a84c1aa764a78865ad91e605714f94aceb WatchSource:0}: Error finding container ec7426e203d77709c23bfbabf8d421a84c1aa764a78865ad91e605714f94aceb: Status 404 returned error can't find the container with id ec7426e203d77709c23bfbabf8d421a84c1aa764a78865ad91e605714f94aceb
Jul 07 14:58:05.171412 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.171315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:05.173821 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.173795    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aeab4723-c9f2-42e8-b3bf-fe0dea37a35b-node-exporter-tls\") pod \"node-exporter-597sm\" (UID: \"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b\") " pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:05.258870 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.258834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lt87x" event={"ID":"2aafa307-bcb5-42de-953b-596ec07d8845","Type":"ContainerStarted","Data":"2bc4df752b7a0a28c4fc152335a79a38a153ae1505f3a36b8eb29b65566a4616"}
Jul 07 14:58:05.259999 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.259973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbkcv" event={"ID":"1f878f43-2b0f-4463-968a-34052c7d9eef","Type":"ContainerStarted","Data":"ec7426e203d77709c23bfbabf8d421a84c1aa764a78865ad91e605714f94aceb"}
Jul 07 14:58:05.261020 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.260998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gqtvk" event={"ID":"cd05c438-977e-42f1-ac3a-10605f8f2d41","Type":"ContainerStarted","Data":"4fdd2d5ead322c4da69d58caac2ceb573a3670c9d668c039dd1a46fd9449a558"}
Jul 07 14:58:05.355219 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:05.355187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-597sm"
Jul 07 14:58:05.367065 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:58:05.367028    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeab4723_c9f2_42e8_b3bf_fe0dea37a35b.slice/crio-6458e257382958a8552890f94343319278b1b0eff82b1425a6c3c373a2a008f4 WatchSource:0}: Error finding container 6458e257382958a8552890f94343319278b1b0eff82b1425a6c3c373a2a008f4: Status 404 returned error can't find the container with id 6458e257382958a8552890f94343319278b1b0eff82b1425a6c3c373a2a008f4
Jul 07 14:58:06.269958 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:06.269908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-597sm" event={"ID":"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b","Type":"ContainerStarted","Data":"6458e257382958a8552890f94343319278b1b0eff82b1425a6c3c373a2a008f4"}
Jul 07 14:58:06.273860 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:06.273770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gqtvk" event={"ID":"cd05c438-977e-42f1-ac3a-10605f8f2d41","Type":"ContainerStarted","Data":"e72d30a19b9089adcb74e3586ccb77296be3c77358d34961bc91311b539b8786"}
Jul 07 14:58:07.223151 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:07.223074    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wc5dm"
Jul 07 14:58:07.281607 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:07.281566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gqtvk" event={"ID":"cd05c438-977e-42f1-ac3a-10605f8f2d41","Type":"ContainerStarted","Data":"92110790f57bac758c61b622fa8ac90f54891d0c0e2595095d8b2c8976113dbf"}
Jul 07 14:58:07.283988 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:07.283928    2575 generic.go:356] "Generic (PLEG): container finished" podID="aeab4723-c9f2-42e8-b3bf-fe0dea37a35b" containerID="6d22f19e79d546efd934798e2b3eb7f5478926805fe5ebd9340e32deec8f3c0e" exitCode=0
Jul 07 14:58:07.284117 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:07.284033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-597sm" event={"ID":"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b","Type":"ContainerDied","Data":"6d22f19e79d546efd934798e2b3eb7f5478926805fe5ebd9340e32deec8f3c0e"}
Jul 07 14:58:07.299818 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:07.299768    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gqtvk" podStartSLOduration=68.263805896 podStartE2EDuration="1m9.299748082s" totalImagesPullingTime="1.035942186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:07.2986866 +0000 UTC m=+68.945225704" watchObservedRunningTime="2026-07-07 14:58:07.299748082 +0000 UTC m=+68.946287186"
Jul 07 14:58:08.290823 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:08.290780    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-597sm" event={"ID":"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b","Type":"ContainerStarted","Data":"50245200929d73c34a303b88147c4d00af4d5f8c2ba280fd69022626a90c7910"}
Jul 07 14:58:08.291271 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:08.290891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-597sm" event={"ID":"aeab4723-c9f2-42e8-b3bf-fe0dea37a35b","Type":"ContainerStarted","Data":"d7080a9ee377648a46845995bc89d17967409aa8f23f20e4784e3018af2fb64b"}
Jul 07 14:58:08.318226 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:08.318175    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-597sm" podStartSLOduration=2.768647186 podStartE2EDuration="4.318157433s" totalImagesPullingTime="1.459372291s" totalInitContainerRuntime="90.137956ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:08.316655941 +0000 UTC m=+69.963195045" watchObservedRunningTime="2026-07-07 14:58:08.318157433 +0000 UTC m=+69.964696535"
Jul 07 14:58:09.196910 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.196877    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"]
Jul 07 14:58:09.199360 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.199336    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.202510 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.202483    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vj2ch\""
Jul 07 14:58:09.202752 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.202735    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 14:58:09.222280 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.222247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"]
Jul 07 14:58:09.323566 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.323530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmfk\" (UniqueName: \"kubernetes.io/projected/f62f4c33-fcea-49b9-931b-da0952304209-kube-api-access-8mmfk\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.324034 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.323596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f62f4c33-fcea-49b9-931b-da0952304209-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.424287 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.424248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f62f4c33-fcea-49b9-931b-da0952304209-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.424484 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.424397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmfk\" (UniqueName: \"kubernetes.io/projected/f62f4c33-fcea-49b9-931b-da0952304209-kube-api-access-8mmfk\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.427154 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.427122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f62f4c33-fcea-49b9-931b-da0952304209-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.437695 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.437664    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmfk\" (UniqueName: \"kubernetes.io/projected/f62f4c33-fcea-49b9-931b-da0952304209-kube-api-access-8mmfk\") pod \"monitoring-plugin-d5dcff7cf-lqxfb\" (UID: \"f62f4c33-fcea-49b9-931b-da0952304209\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:09.525475 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:09.525437    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:10.112676 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.112643    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"]
Jul 07 14:58:10.116024 ip-10-0-128-215 kubenswrapper[2575]: W0707 14:58:10.115997    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62f4c33_fcea_49b9_931b_da0952304209.slice/crio-58dd466cfddb87054fedb80d7dcd275f0ac478daa477595cf68038e5121e018f WatchSource:0}: Error finding container 58dd466cfddb87054fedb80d7dcd275f0ac478daa477595cf68038e5121e018f: Status 404 returned error can't find the container with id 58dd466cfddb87054fedb80d7dcd275f0ac478daa477595cf68038e5121e018f
Jul 07 14:58:10.299672 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.299634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lt87x" event={"ID":"2aafa307-bcb5-42de-953b-596ec07d8845","Type":"ContainerStarted","Data":"e22354f4e7bfae33957a85eafc98616187a44b3ab7e11312b710ef740b499de7"}
Jul 07 14:58:10.300780 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.300752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb" event={"ID":"f62f4c33-fcea-49b9-931b-da0952304209","Type":"ContainerStarted","Data":"58dd466cfddb87054fedb80d7dcd275f0ac478daa477595cf68038e5121e018f"}
Jul 07 14:58:10.302082 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.302058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbkcv" event={"ID":"1f878f43-2b0f-4463-968a-34052c7d9eef","Type":"ContainerStarted","Data":"11f4069b08ab9ea1cfc2d2d1c45a4b3e80cfbe5c71b0a53a1dd9fb593962db3c"}
Jul 07 14:58:10.302200 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.302184    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 14:58:10.320595 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.320545    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lt87x" podStartSLOduration=66.343684056 podStartE2EDuration="1m11.320527113s" totalImagesPullingTime="4.976843057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:10.319083657 +0000 UTC m=+71.965622757" watchObservedRunningTime="2026-07-07 14:58:10.320527113 +0000 UTC m=+71.967066216"
Jul 07 14:58:10.344901 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:10.344839    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rbkcv" podStartSLOduration=66.408338098 podStartE2EDuration="1m11.344820483s" totalImagesPullingTime="4.936482385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:10.34417463 +0000 UTC m=+71.990713734" watchObservedRunningTime="2026-07-07 14:58:10.344820483 +0000 UTC m=+71.991359588"
Jul 07 14:58:12.310914 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:12.310878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb" event={"ID":"f62f4c33-fcea-49b9-931b-da0952304209","Type":"ContainerStarted","Data":"76ab5ced92e519eb1cc398eb73dd5e2baf0ffb17040fe48ca463f6d596b7630f"}
Jul 07 14:58:12.311324 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:12.311063    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:12.315825 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:12.315798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb"
Jul 07 14:58:12.328923 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:12.328872    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-lqxfb" podStartSLOduration=1.479363876 podStartE2EDuration="3.328852854s" totalImagesPullingTime="1.849488978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:12.327238318 +0000 UTC m=+73.973777421" watchObservedRunningTime="2026-07-07 14:58:12.328852854 +0000 UTC m=+73.975391958"
Jul 07 14:58:14.276063 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:14.276024    2575 patch_prober.go:28] interesting pod/image-registry-5cc7df4966-ljnd4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 14:58:14.276444 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:14.276078    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4" podUID="55a9ebce-7148-4feb-93a7-cb0158b5754e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:58:17.222034 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:17.221840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cc7df4966-ljnd4"
Jul 07 14:58:41.309564 ip-10-0-128-215 kubenswrapper[2575]: I0707 14:58:41.309527    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rbkcv"
Jul 07 15:01:05.450578 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.450546    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-n99rz"]
Jul 07 15:01:05.452841 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.452825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.455951 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.455910    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 07 15:01:05.456950 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.456917    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 07 15:01:05.456950 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.456927    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 07 15:01:05.457106 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.457041    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 07 15:01:05.457237 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.457223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-f8s48\""
Jul 07 15:01:05.465894 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.465869    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-n99rz"]
Jul 07 15:01:05.493330 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.493296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9zt\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-kube-api-access-sz9zt\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.493511 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.493359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-certificates\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.594728 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.594693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-certificates\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.594728 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.594736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9zt\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-kube-api-access-sz9zt\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.597424 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.597390    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-certificates\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.607812 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.607783    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9zt\" (UniqueName: \"kubernetes.io/projected/60edcd91-d10e-430d-9aec-b20d514d6687-kube-api-access-sz9zt\") pod \"keda-admission-5fbd5c4644-n99rz\" (UID: \"60edcd91-d10e-430d-9aec-b20d514d6687\") " pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.763121 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.763081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:05.912879 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:05.912840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-n99rz"]
Jul 07 15:01:05.917554 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:01:05.917528    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60edcd91_d10e_430d_9aec_b20d514d6687.slice/crio-8ccdb1cc0181690363c18ea871f446703d9f23b4f6fdbf24852b0e73e0dbfb56 WatchSource:0}: Error finding container 8ccdb1cc0181690363c18ea871f446703d9f23b4f6fdbf24852b0e73e0dbfb56: Status 404 returned error can't find the container with id 8ccdb1cc0181690363c18ea871f446703d9f23b4f6fdbf24852b0e73e0dbfb56
Jul 07 15:01:06.906297 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:06.906228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz" event={"ID":"60edcd91-d10e-430d-9aec-b20d514d6687","Type":"ContainerStarted","Data":"8ccdb1cc0181690363c18ea871f446703d9f23b4f6fdbf24852b0e73e0dbfb56"}
Jul 07 15:01:08.918193 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:08.918161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz" event={"ID":"60edcd91-d10e-430d-9aec-b20d514d6687","Type":"ContainerStarted","Data":"88a1318e7c58c9e370dccf2c3a83e323d0f3a539cd696a5beb044d5def2a45d1"}
Jul 07 15:01:08.918659 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:08.918214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:08.938269 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:08.938205    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz" podStartSLOduration=1.4752631379999999 podStartE2EDuration="3.938183132s" totalImagesPullingTime="2.462919994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:01:08.937626644 +0000 UTC m=+250.584165751" watchObservedRunningTime="2026-07-07 15:01:08.938183132 +0000 UTC m=+250.584722236"
Jul 07 15:01:29.920177 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:29.920097    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-n99rz"
Jul 07 15:01:58.740631 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:01:58.740606    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 15:02:14.708605 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.708568    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"]
Jul 07 15:02:14.710947 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.710922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.714111 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.714085    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 07 15:02:14.714265 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.714150    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 15:02:14.714265 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.714202    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 15:02:14.715068 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.715052    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zk2cv\""
Jul 07 15:02:14.730350 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.730319    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"]
Jul 07 15:02:14.805041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.805007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vs6z\" (UniqueName: \"kubernetes.io/projected/a03ce0de-8345-4147-8df5-0b660f383fd8-kube-api-access-4vs6z\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.805206 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.805059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03ce0de-8345-4147-8df5-0b660f383fd8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.906209 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.906172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vs6z\" (UniqueName: \"kubernetes.io/projected/a03ce0de-8345-4147-8df5-0b660f383fd8-kube-api-access-4vs6z\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.906395 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.906229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03ce0de-8345-4147-8df5-0b660f383fd8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.908798 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.908767    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a03ce0de-8345-4147-8df5-0b660f383fd8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:14.919408 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:14.919378    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vs6z\" (UniqueName: \"kubernetes.io/projected/a03ce0de-8345-4147-8df5-0b660f383fd8-kube-api-access-4vs6z\") pod \"llmisvc-controller-manager-67d95fdc48-qnmkk\" (UID: \"a03ce0de-8345-4147-8df5-0b660f383fd8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:15.021797 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:15.021763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:15.169785 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:15.169754    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"]
Jul 07 15:02:15.173371 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:02:15.173344    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda03ce0de_8345_4147_8df5_0b660f383fd8.slice/crio-db815240f10cdee291bad18d48ed6a1419ae944facb7cb718a2e732340b4c43b WatchSource:0}: Error finding container db815240f10cdee291bad18d48ed6a1419ae944facb7cb718a2e732340b4c43b: Status 404 returned error can't find the container with id db815240f10cdee291bad18d48ed6a1419ae944facb7cb718a2e732340b4c43b
Jul 07 15:02:15.175149 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:15.175132    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:02:16.136720 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:16.136685    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk" event={"ID":"a03ce0de-8345-4147-8df5-0b660f383fd8","Type":"ContainerStarted","Data":"db815240f10cdee291bad18d48ed6a1419ae944facb7cb718a2e732340b4c43b"}
Jul 07 15:02:23.163786 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:23.163751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk" event={"ID":"a03ce0de-8345-4147-8df5-0b660f383fd8","Type":"ContainerStarted","Data":"81398b3fbbfe8956eca706c7628bd378cfb9a8cac57d08352272da8718db681c"}
Jul 07 15:02:23.164283 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:23.163867    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:02:23.182694 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:23.182638    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk" podStartSLOduration=1.379751252 podStartE2EDuration="9.182621338s" totalImagesPullingTime="7.802870086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:02:23.181575715 +0000 UTC m=+324.828114818" watchObservedRunningTime="2026-07-07 15:02:23.182621338 +0000 UTC m=+324.829160441"
Jul 07 15:02:54.169790 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:02:54.169713    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-qnmkk"
Jul 07 15:03:30.206770 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.206725    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-fc8zs"]
Jul 07 15:03:30.208868 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.208850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.211890 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.211868    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 15:03:30.212036 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.211912    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xw4ml\""
Jul 07 15:03:30.221121 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.221097    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-fc8zs"]
Jul 07 15:03:30.374437 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.374388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb894d-c153-4c96-8081-e7834231378a-tls-certs\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.374620 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.374449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6kf\" (UniqueName: \"kubernetes.io/projected/38eb894d-c153-4c96-8081-e7834231378a-kube-api-access-tl6kf\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.475898 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.475793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb894d-c153-4c96-8081-e7834231378a-tls-certs\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.475898 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.475848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6kf\" (UniqueName: \"kubernetes.io/projected/38eb894d-c153-4c96-8081-e7834231378a-kube-api-access-tl6kf\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.478380 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.478349    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38eb894d-c153-4c96-8081-e7834231378a-tls-certs\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.485249 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.485221    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6kf\" (UniqueName: \"kubernetes.io/projected/38eb894d-c153-4c96-8081-e7834231378a-kube-api-access-tl6kf\") pod \"model-serving-api-fd65d7d6-fc8zs\" (UID: \"38eb894d-c153-4c96-8081-e7834231378a\") " pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.518959 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.518898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:30.656058 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:30.656024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-fc8zs"]
Jul 07 15:03:30.659085 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:03:30.659056    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eb894d_c153_4c96_8081_e7834231378a.slice/crio-b8af5f55835a9d8d4d2ab8239185e645731574ef7409b4a9c45af79eca05e161 WatchSource:0}: Error finding container b8af5f55835a9d8d4d2ab8239185e645731574ef7409b4a9c45af79eca05e161: Status 404 returned error can't find the container with id b8af5f55835a9d8d4d2ab8239185e645731574ef7409b4a9c45af79eca05e161
Jul 07 15:03:31.391599 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:31.391559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-fc8zs" event={"ID":"38eb894d-c153-4c96-8081-e7834231378a","Type":"ContainerStarted","Data":"b8af5f55835a9d8d4d2ab8239185e645731574ef7409b4a9c45af79eca05e161"}
Jul 07 15:03:34.403155 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:34.403117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-fc8zs" event={"ID":"38eb894d-c153-4c96-8081-e7834231378a","Type":"ContainerStarted","Data":"08a34cfac18182b1474c6cd81487acbbe9ac29c90504b19b9978f5b35c6bf276"}
Jul 07 15:03:34.403549 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:34.403218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:45.411765 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:45.411733    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-fc8zs"
Jul 07 15:03:45.439845 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:03:45.439785    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-fc8zs" podStartSLOduration=12.306286998 podStartE2EDuration="15.439749967s" totalImagesPullingTime="3.133462969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:03:34.433663665 +0000 UTC m=+396.080202768" watchObservedRunningTime="2026-07-07 15:03:45.439749967 +0000 UTC m=+407.086289081"
Jul 07 15:05:20.009205 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.009171    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:05:20.011570 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.011554    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.014230 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.014201    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f85ce-kube-rbac-proxy-sar-config\""
Jul 07 15:05:20.014448 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.014429    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 15:05:20.014688 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.014674    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f85ce-predictor-serving-cert\""
Jul 07 15:05:20.015306 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.015287    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gzwhn\""
Jul 07 15:05:20.015378 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.015287    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 15:05:20.023433 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.023409    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:05:20.095956 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.095907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.096179 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.096004    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkvk\" (UniqueName: \"kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.096179 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.096114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.196867 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.196825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.197090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.196898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkvk\" (UniqueName: \"kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.197090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.196972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.197497 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.197469    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.199513 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.199496    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.211773 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.211738    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkvk\" (UniqueName: \"kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk\") pod \"error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") " pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.323794 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.323697    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:20.413517 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.413299    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:05:20.417471 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.416576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.422039 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.421577    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 07 15:05:20.422039 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.421853    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 15:05:20.422039 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.422005    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 07 15:05:20.422317 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.422155    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 15:05:20.432183 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.432159    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:05:20.473465 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.473433    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:05:20.476765 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:05:20.476739    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f87be0_c8dd_48dc_b20e_d05e5c34750b.slice/crio-8f77d37cbf0586e52fb66e815d2bcd5e9c00e1383b30ffa7ce486ba982dd7ff9 WatchSource:0}: Error finding container 8f77d37cbf0586e52fb66e815d2bcd5e9c00e1383b30ffa7ce486ba982dd7ff9: Status 404 returned error can't find the container with id 8f77d37cbf0586e52fb66e815d2bcd5e9c00e1383b30ffa7ce486ba982dd7ff9
Jul 07 15:05:20.499951 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.499907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.500109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.499985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.500109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.500015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx64n\" (UniqueName: \"kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.500109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.500042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.500234 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.500110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.600984 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.600880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.600984 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.600920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.600984 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.600979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.601237 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.601022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx64n\" (UniqueName: \"kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.601237 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.601065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.602022 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.601990    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.602144 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.602027    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.602217 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.602188    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.606524 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.604701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.610912 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.610885    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx64n\" (UniqueName: \"kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n\") pod \"isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.730524 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.730468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:20.772892 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.772856    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerStarted","Data":"8f77d37cbf0586e52fb66e815d2bcd5e9c00e1383b30ffa7ce486ba982dd7ff9"}
Jul 07 15:05:20.862304 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:20.862223    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:05:20.864973 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:05:20.864929    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a0b1b1_74a5_456d_a03f_d13fbb47ece7.slice/crio-32c0286f907d54e339cf2c7a4d953e79bd67a32fe285db4db9ed0ef9fb83648c WatchSource:0}: Error finding container 32c0286f907d54e339cf2c7a4d953e79bd67a32fe285db4db9ed0ef9fb83648c: Status 404 returned error can't find the container with id 32c0286f907d54e339cf2c7a4d953e79bd67a32fe285db4db9ed0ef9fb83648c
Jul 07 15:05:21.785715 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:21.785674    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerStarted","Data":"32c0286f907d54e339cf2c7a4d953e79bd67a32fe285db4db9ed0ef9fb83648c"}
Jul 07 15:05:33.840781 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:33.840741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerStarted","Data":"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"}
Jul 07 15:05:33.842384 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:33.842356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerStarted","Data":"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"}
Jul 07 15:05:36.855542 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:36.855499    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerStarted","Data":"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"}
Jul 07 15:05:36.855994 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:36.855594    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:36.855994 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:36.855656    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:36.856843 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:36.856818    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:05:36.876389 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:36.876346    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podStartSLOduration=2.262436475 podStartE2EDuration="17.876331665s" totalImagesPullingTime="15.61389519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:05:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:05:36.875139182 +0000 UTC m=+518.521678285" watchObservedRunningTime="2026-07-07 15:05:36.876331665 +0000 UTC m=+518.522870768"
Jul 07 15:05:37.860465 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:37.860430    2575 generic.go:356] "Generic (PLEG): container finished" podID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerID="ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15" exitCode=0
Jul 07 15:05:37.860915 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:37.860504    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerDied","Data":"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"}
Jul 07 15:05:37.860915 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:37.860857    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:05:42.866565 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:42.866528    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:05:42.867204 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:42.867176    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:05:44.893389 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:44.893353    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerStarted","Data":"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"}
Jul 07 15:05:44.893876 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:44.893424    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerStarted","Data":"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"}
Jul 07 15:05:44.893876 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:44.893731    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:44.917411 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:44.917322    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podStartSLOduration=2.169829582 podStartE2EDuration="24.917308648s" totalImagesPullingTime="18.93625643s" totalInitContainerRuntime="3.811222636s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:05:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:05:44.916790325 +0000 UTC m=+526.563329435" watchObservedRunningTime="2026-07-07 15:05:44.917308648 +0000 UTC m=+526.563847751"
Jul 07 15:05:45.897316 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:45.897280    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:45.898480 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:45.898450    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:05:46.902846 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:46.902804    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:05:51.907680 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:51.907594    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:05:51.908280 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:51.908122    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:05:52.867832 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:05:52.867790    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:06:01.908279 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:01.908242    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:02.867742 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:02.867706    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:06:11.908410 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:11.908349    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:12.867266 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:12.867225    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 15:06:21.908159 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:21.908120    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:22.868100 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:22.868060    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:06:31.908892 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:31.908849    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:41.908286 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:41.908246    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:51.909030 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:51.908984    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:54.266770 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.266731    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:06:54.269145 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.269129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.274479 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.274449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c9c9-predictor-serving-cert\""
Jul 07 15:06:54.274677 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.274457    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c9c9-kube-rbac-proxy-sar-config\""
Jul 07 15:06:54.288846 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.288811    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:06:54.376888 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.376839    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.377097 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.376906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.377097 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.376988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpls\" (UniqueName: \"kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.473985 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.473929    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:06:54.474222 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.474182    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container" containerID="cri-o://eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01" gracePeriod=30
Jul 07 15:06:54.474333 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.474278    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kube-rbac-proxy" containerID="cri-o://cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c" gracePeriod=30
Jul 07 15:06:54.478086 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.478058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.478194 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.478110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpls\" (UniqueName: \"kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.478194 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.478173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.478279 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:06:54.478206    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-serving-cert: secret "error-404-isvc-3c9c9-predictor-serving-cert" not found
Jul 07 15:06:54.478279 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:06:54.478268    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls podName:31c08cbd-5315-4979-8937-4559fee41cf0 nodeName:}" failed. No retries permitted until 2026-07-07 15:06:54.978250174 +0000 UTC m=+596.624789258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls") pod "error-404-isvc-3c9c9-predictor-7d847f679-zghqk" (UID: "31c08cbd-5315-4979-8937-4559fee41cf0") : secret "error-404-isvc-3c9c9-predictor-serving-cert" not found
Jul 07 15:06:54.479019 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.478874    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.490476 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.490451    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpls\" (UniqueName: \"kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.983544 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.983453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:54.985966 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:54.985921    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") pod \"error-404-isvc-3c9c9-predictor-7d847f679-zghqk\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") " pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:55.151010 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:55.150969    2575 generic.go:356] "Generic (PLEG): container finished" podID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerID="cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c" exitCode=2
Jul 07 15:06:55.151164 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:55.151044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerDied","Data":"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"}
Jul 07 15:06:55.179470 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:55.179440    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:55.325355 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:55.325330    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:06:55.327822 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:06:55.327796    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c08cbd_5315_4979_8937_4559fee41cf0.slice/crio-57e0be39465346f4224accc7123f94bfb006c5b81ed7f0f04fdab3f7f3017945 WatchSource:0}: Error finding container 57e0be39465346f4224accc7123f94bfb006c5b81ed7f0f04fdab3f7f3017945: Status 404 returned error can't find the container with id 57e0be39465346f4224accc7123f94bfb006c5b81ed7f0f04fdab3f7f3017945
Jul 07 15:06:56.156722 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:56.156685    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerStarted","Data":"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"}
Jul 07 15:06:56.156950 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:56.156740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerStarted","Data":"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"}
Jul 07 15:06:56.156950 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:56.156764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerStarted","Data":"57e0be39465346f4224accc7123f94bfb006c5b81ed7f0f04fdab3f7f3017945"}
Jul 07 15:06:56.156950 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:56.156841    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:56.177804 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:56.177752    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podStartSLOduration=2.177734489 podStartE2EDuration="2.177734489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:06:56.17562956 +0000 UTC m=+597.822168662" watchObservedRunningTime="2026-07-07 15:06:56.177734489 +0000 UTC m=+597.824273592"
Jul 07 15:06:57.161453 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:57.161415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:06:57.162774 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:57.162744    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:06:57.920206 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:57.920175    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:06:58.011643 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.011616    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbkvk\" (UniqueName: \"kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk\") pod \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") "
Jul 07 15:06:58.011801 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.011683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config\") pod \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") "
Jul 07 15:06:58.011801 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.011720    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls\") pod \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\" (UID: \"42f87be0-c8dd-48dc-b20e-d05e5c34750b\") "
Jul 07 15:06:58.012079 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.012054    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42f87be0-c8dd-48dc-b20e-d05e5c34750b" (UID: "42f87be0-c8dd-48dc-b20e-d05e5c34750b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:06:58.014067 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.014046    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42f87be0-c8dd-48dc-b20e-d05e5c34750b" (UID: "42f87be0-c8dd-48dc-b20e-d05e5c34750b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:06:58.014134 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.014111    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk" (OuterVolumeSpecName: "kube-api-access-vbkvk") pod "42f87be0-c8dd-48dc-b20e-d05e5c34750b" (UID: "42f87be0-c8dd-48dc-b20e-d05e5c34750b"). InnerVolumeSpecName "kube-api-access-vbkvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:06:58.112337 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.112303    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbkvk\" (UniqueName: \"kubernetes.io/projected/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-api-access-vbkvk\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:06:58.112337 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.112333    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42f87be0-c8dd-48dc-b20e-d05e5c34750b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:06:58.112337 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.112343    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42f87be0-c8dd-48dc-b20e-d05e5c34750b-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:06:58.166367 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.166335    2575 generic.go:356] "Generic (PLEG): container finished" podID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerID="eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01" exitCode=0
Jul 07 15:06:58.166825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.166388    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"
Jul 07 15:06:58.166825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.166420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerDied","Data":"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"}
Jul 07 15:06:58.166825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.166495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" event={"ID":"42f87be0-c8dd-48dc-b20e-d05e5c34750b","Type":"ContainerDied","Data":"8f77d37cbf0586e52fb66e815d2bcd5e9c00e1383b30ffa7ce486ba982dd7ff9"}
Jul 07 15:06:58.166825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.166540    2575 scope.go:117] "RemoveContainer" containerID="cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"
Jul 07 15:06:58.167223 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.167021    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:06:58.175444 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.175427    2575 scope.go:117] "RemoveContainer" containerID="eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"
Jul 07 15:06:58.182814 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.182799    2575 scope.go:117] "RemoveContainer" containerID="cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"
Jul 07 15:06:58.183129 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:06:58.183106    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c\": container with ID starting with cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c not found: ID does not exist" containerID="cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"
Jul 07 15:06:58.183216 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.183142    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c"} err="failed to get container status \"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c\": rpc error: code = NotFound desc = could not find container \"cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c\": container with ID starting with cdba8681d5509ef94b542ba8b34bd54c6d03c0664aae227db9b2e67366b6be2c not found: ID does not exist"
Jul 07 15:06:58.183216 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.183181    2575 scope.go:117] "RemoveContainer" containerID="eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"
Jul 07 15:06:58.183452 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:06:58.183433    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01\": container with ID starting with eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01 not found: ID does not exist" containerID="eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"
Jul 07 15:06:58.183495 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.183460    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01"} err="failed to get container status \"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01\": rpc error: code = NotFound desc = could not find container \"eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01\": container with ID starting with eeed72d2ced0a957b64f5538cde80ba791e7bec052ef05588c85e5ce88111b01 not found: ID does not exist"
Jul 07 15:06:58.189310 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.189283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:06:58.192603 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.192580    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg"]
Jul 07 15:06:58.862131 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.862080    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f85ce-predictor-859cc5b5d7-qbbgg" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 07 15:06:58.916672 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.916635    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 15:06:58.919269 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:06:58.919243    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" path="/var/lib/kubelet/pods/42f87be0-c8dd-48dc-b20e-d05e5c34750b/volumes"
Jul 07 15:07:03.171557 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:03.171528    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:07:03.172175 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:03.172146    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:07:08.919232 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:08.919208    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:07:13.172249 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:13.172204    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:07:23.172891 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:23.172811    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:07:29.912314 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:29.912279    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:07:29.912869 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:29.912643    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container" containerID="cri-o://e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a" gracePeriod=30
Jul 07 15:07:29.912869 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:29.912745    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kube-rbac-proxy" containerID="cri-o://f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1" gracePeriod=30
Jul 07 15:07:30.145345 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145308    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:07:30.145719 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145704    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container"
Jul 07 15:07:30.145719 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145720    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container"
Jul 07 15:07:30.145847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145738    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kube-rbac-proxy"
Jul 07 15:07:30.145847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145744    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kube-rbac-proxy"
Jul 07 15:07:30.145847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145791    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kube-rbac-proxy"
Jul 07 15:07:30.145847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.145801    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f87be0-c8dd-48dc-b20e-d05e5c34750b" containerName="kserve-container"
Jul 07 15:07:30.148088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.148070    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.150478 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.150457    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8996c-predictor-serving-cert\""
Jul 07 15:07:30.150596 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.150461    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8996c-kube-rbac-proxy-sar-config\""
Jul 07 15:07:30.153309 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.153284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.153439 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.153345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.153439 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.153404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfkf\" (UniqueName: \"kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.158000 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.157921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:07:30.253760 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.253719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfkf\" (UniqueName: \"kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.254074 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.253789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.254074 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.253862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.254680 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.254611    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.256390 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.256367    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.263681 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.263641    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfkf\" (UniqueName: \"kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf\") pod \"error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") " pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.282685 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.282647    2575 generic.go:356] "Generic (PLEG): container finished" podID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerID="f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1" exitCode=2
Jul 07 15:07:30.282848 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.282720    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerDied","Data":"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"}
Jul 07 15:07:30.460321 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.460278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:30.605662 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:30.605601    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:07:30.607888 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:07:30.607857    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196ffcbf_7496_4f27_8599_d1b6d11fe543.slice/crio-ff27d4b46c06c597ec216968eb4c24753c126bc1f41947144e9b2e65528a99a6 WatchSource:0}: Error finding container ff27d4b46c06c597ec216968eb4c24753c126bc1f41947144e9b2e65528a99a6: Status 404 returned error can't find the container with id ff27d4b46c06c597ec216968eb4c24753c126bc1f41947144e9b2e65528a99a6
Jul 07 15:07:31.288664 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.288624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerStarted","Data":"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"}
Jul 07 15:07:31.289203 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.288692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerStarted","Data":"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"}
Jul 07 15:07:31.289203 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.288726    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerStarted","Data":"ff27d4b46c06c597ec216968eb4c24753c126bc1f41947144e9b2e65528a99a6"}
Jul 07 15:07:31.289203 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.288757    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:31.309197 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.309144    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podStartSLOduration=1.309127991 podStartE2EDuration="1.309127991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:31.308518971 +0000 UTC m=+632.955058077" watchObservedRunningTime="2026-07-07 15:07:31.309127991 +0000 UTC m=+632.955667094"
Jul 07 15:07:31.903913 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:31.903867    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 15:07:32.293745 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:32.293661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:32.295360 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:32.295025    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:07:33.172103 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:33.172063    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 15:07:33.297732 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:33.297698    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:07:34.960211 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.960182    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:07:34.994880 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.994846    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config\") pod \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") "
Jul 07 15:07:34.995109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.994981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx64n\" (UniqueName: \"kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n\") pod \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") "
Jul 07 15:07:34.995109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location\") pod \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") "
Jul 07 15:07:34.995109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995045    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert\") pod \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") "
Jul 07 15:07:34.995109 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995075    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls\") pod \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\" (UID: \"32a0b1b1-74a5-456d-a03f-d13fbb47ece7\") "
Jul 07 15:07:34.995426 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995353    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32a0b1b1-74a5-456d-a03f-d13fbb47ece7" (UID: "32a0b1b1-74a5-456d-a03f-d13fbb47ece7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 15:07:34.995677 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995645    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32a0b1b1-74a5-456d-a03f-d13fbb47ece7" (UID: "32a0b1b1-74a5-456d-a03f-d13fbb47ece7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:07:34.995779 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.995697    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32a0b1b1-74a5-456d-a03f-d13fbb47ece7" (UID: "32a0b1b1-74a5-456d-a03f-d13fbb47ece7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:07:34.997646 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.997616    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32a0b1b1-74a5-456d-a03f-d13fbb47ece7" (UID: "32a0b1b1-74a5-456d-a03f-d13fbb47ece7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:07:34.997727 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:34.997634    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n" (OuterVolumeSpecName: "kube-api-access-vx64n") pod "32a0b1b1-74a5-456d-a03f-d13fbb47ece7" (UID: "32a0b1b1-74a5-456d-a03f-d13fbb47ece7"). InnerVolumeSpecName "kube-api-access-vx64n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:07:35.096167 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.096067    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx64n\" (UniqueName: \"kubernetes.io/projected/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-api-access-vx64n\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:07:35.096167 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.096103    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:07:35.096167 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.096112    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-cabundle-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:07:35.096167 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.096122    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:07:35.096167 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.096130    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32a0b1b1-74a5-456d-a03f-d13fbb47ece7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:07:35.307135 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.307100    2575 generic.go:356] "Generic (PLEG): container finished" podID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerID="e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a" exitCode=0
Jul 07 15:07:35.307320 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.307166    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"
Jul 07 15:07:35.307320 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.307188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerDied","Data":"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"}
Jul 07 15:07:35.307320 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.307246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b" event={"ID":"32a0b1b1-74a5-456d-a03f-d13fbb47ece7","Type":"ContainerDied","Data":"32c0286f907d54e339cf2c7a4d953e79bd67a32fe285db4db9ed0ef9fb83648c"}
Jul 07 15:07:35.307320 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.307274    2575 scope.go:117] "RemoveContainer" containerID="f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"
Jul 07 15:07:35.316260 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.316241    2575 scope.go:117] "RemoveContainer" containerID="e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"
Jul 07 15:07:35.323892 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.323872    2575 scope.go:117] "RemoveContainer" containerID="ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"
Jul 07 15:07:35.331773 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.331745    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:07:35.332062 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332039    2575 scope.go:117] "RemoveContainer" containerID="f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"
Jul 07 15:07:35.332350 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:07:35.332331    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1\": container with ID starting with f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1 not found: ID does not exist" containerID="f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"
Jul 07 15:07:35.332389 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332363    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1"} err="failed to get container status \"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1\": rpc error: code = NotFound desc = could not find container \"f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1\": container with ID starting with f3b9e68abc4946a0d4e97e7c26a24ac3650a8debfa27823b3c396f7554ad01a1 not found: ID does not exist"
Jul 07 15:07:35.332389 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332383    2575 scope.go:117] "RemoveContainer" containerID="e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"
Jul 07 15:07:35.332634 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:07:35.332615    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a\": container with ID starting with e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a not found: ID does not exist" containerID="e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"
Jul 07 15:07:35.332690 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332639    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a"} err="failed to get container status \"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a\": rpc error: code = NotFound desc = could not find container \"e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a\": container with ID starting with e4094232c588f2625aca8835f470bbf0e40ee07272d5f7cb2070d6b3e863ea4a not found: ID does not exist"
Jul 07 15:07:35.332690 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332655    2575 scope.go:117] "RemoveContainer" containerID="ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"
Jul 07 15:07:35.332877 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:07:35.332859    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15\": container with ID starting with ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15 not found: ID does not exist" containerID="ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"
Jul 07 15:07:35.332983 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.332881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15"} err="failed to get container status \"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15\": rpc error: code = NotFound desc = could not find container \"ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15\": container with ID starting with ef230335592427e7ef892efa4dc7a43733103c75871d8ba10067424d67c97d15 not found: ID does not exist"
Jul 07 15:07:35.335568 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:35.335545    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-767d6d7f5d-xkw4b"]
Jul 07 15:07:36.920048 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:36.920014    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" path="/var/lib/kubelet/pods/32a0b1b1-74a5-456d-a03f-d13fbb47ece7/volumes"
Jul 07 15:07:38.302205 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:38.302179    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:07:38.302697 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:38.302673    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:07:43.173088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:43.173057    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:07:48.303041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:48.302988    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:07:54.111478 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111440    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111722    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="storage-initializer"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111732    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="storage-initializer"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111747    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kube-rbac-proxy"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111752    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kube-rbac-proxy"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111759    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111765    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111811    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kube-rbac-proxy"
Jul 07 15:07:54.111897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.111819    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a0b1b1-74a5-456d-a03f-d13fbb47ece7" containerName="kserve-container"
Jul 07 15:07:54.113649 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.113634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.116588 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.116566    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3c9c9-kube-rbac-proxy-sar-config\""
Jul 07 15:07:54.116686 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.116613    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3c9c9-serving-cert\""
Jul 07 15:07:54.123284 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.123259    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:07:54.243163 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.243120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.243355 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.243188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.344248 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.344209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.344436 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.344266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.344871 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.344848    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.346886 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.346869    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls\") pod \"switch-graph-3c9c9-7bd8bf8867-hjp6q\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") " pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.424926 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.424845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:54.558817 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.558784    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:07:54.561834 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:07:54.561807    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86140778_5169_4d13_bad6_258d806b5763.slice/crio-d7ff89b71dc1338a6e62ab8538f98e33f64c519b1d78d4eda0d93b13a0eeb549 WatchSource:0}: Error finding container d7ff89b71dc1338a6e62ab8538f98e33f64c519b1d78d4eda0d93b13a0eeb549: Status 404 returned error can't find the container with id d7ff89b71dc1338a6e62ab8538f98e33f64c519b1d78d4eda0d93b13a0eeb549
Jul 07 15:07:54.563741 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:54.563725    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 15:07:55.387230 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:55.387196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" event={"ID":"86140778-5169-4d13-bad6-258d806b5763","Type":"ContainerStarted","Data":"d7ff89b71dc1338a6e62ab8538f98e33f64c519b1d78d4eda0d93b13a0eeb549"}
Jul 07 15:07:57.403381 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:57.403337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" event={"ID":"86140778-5169-4d13-bad6-258d806b5763","Type":"ContainerStarted","Data":"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"}
Jul 07 15:07:57.403778 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:57.403441    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:07:57.425575 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:57.425522    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podStartSLOduration=1.1287320570000001 podStartE2EDuration="3.425507585s" totalImagesPullingTime="2.296775528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:57.424249262 +0000 UTC m=+659.070788364" watchObservedRunningTime="2026-07-07 15:07:57.425507585 +0000 UTC m=+659.072046705"
Jul 07 15:07:58.302827 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:07:58.302788    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:08:03.413825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:08:03.413797    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:08:08.302806 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:08:08.302771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 07 15:08:18.303928 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:08:18.303894    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:16:08.866143 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:08.866110    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:16:08.866654 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:08.866309    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" containerID="cri-o://2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453" gracePeriod=30
Jul 07 15:16:09.296737 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.296697    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:16:09.299332 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.299315    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.301899 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.301872    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-542f9-predictor-serving-cert\""
Jul 07 15:16:09.302067 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.301964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-542f9-kube-rbac-proxy-sar-config\""
Jul 07 15:16:09.312451 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.312421    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:16:09.335609 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.335556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.335809 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.335669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.335809 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.335726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9dz\" (UniqueName: \"kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.352731 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.352697    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:16:09.352981 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.352948    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container" containerID="cri-o://a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795" gracePeriod=30
Jul 07 15:16:09.353116 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.352995    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kube-rbac-proxy" containerID="cri-o://ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac" gracePeriod=30
Jul 07 15:16:09.436842 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.436799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9dz\" (UniqueName: \"kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.437090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.436866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.437090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.436904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.437622 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.437602    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.439567 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.439542    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.445688 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.445660    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9dz\" (UniqueName: \"kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz\") pod \"error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") " pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.610190 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.610101    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:09.751030 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:09.751000    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:16:09.754343 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:16:09.754312    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd396cf3_880b_4780_8cfa_c7589254de62.slice/crio-4cc2382275067aaa46a0126c09fd6ffef8d4979504b4bac80706b3954b2a2737 WatchSource:0}: Error finding container 4cc2382275067aaa46a0126c09fd6ffef8d4979504b4bac80706b3954b2a2737: Status 404 returned error can't find the container with id 4cc2382275067aaa46a0126c09fd6ffef8d4979504b4bac80706b3954b2a2737
Jul 07 15:16:10.168797 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.168693    2575 generic.go:356] "Generic (PLEG): container finished" podID="31c08cbd-5315-4979-8937-4559fee41cf0" containerID="ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac" exitCode=2
Jul 07 15:16:10.168797 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.168762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerDied","Data":"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"}
Jul 07 15:16:10.170603 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.170571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerStarted","Data":"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"}
Jul 07 15:16:10.170767 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.170622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerStarted","Data":"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"}
Jul 07 15:16:10.170767 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.170644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerStarted","Data":"4cc2382275067aaa46a0126c09fd6ffef8d4979504b4bac80706b3954b2a2737"}
Jul 07 15:16:10.170767 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.170676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:10.190029 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:10.189971    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podStartSLOduration=1.189930099 podStartE2EDuration="1.189930099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:16:10.189827391 +0000 UTC m=+1151.836366493" watchObservedRunningTime="2026-07-07 15:16:10.189930099 +0000 UTC m=+1151.836469194"
Jul 07 15:16:11.174421 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:11.174389    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:11.175851 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:11.175816    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:12.179091 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.179051    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:12.914434 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.914408    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:16:12.963849 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.963756    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") pod \"31c08cbd-5315-4979-8937-4559fee41cf0\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") "
Jul 07 15:16:12.963849 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.963791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config\") pod \"31c08cbd-5315-4979-8937-4559fee41cf0\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") "
Jul 07 15:16:12.964050 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.963894    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlpls\" (UniqueName: \"kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls\") pod \"31c08cbd-5315-4979-8937-4559fee41cf0\" (UID: \"31c08cbd-5315-4979-8937-4559fee41cf0\") "
Jul 07 15:16:12.964258 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.964223    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31c08cbd-5315-4979-8937-4559fee41cf0" (UID: "31c08cbd-5315-4979-8937-4559fee41cf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:12.966027 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.966004    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls" (OuterVolumeSpecName: "kube-api-access-zlpls") pod "31c08cbd-5315-4979-8937-4559fee41cf0" (UID: "31c08cbd-5315-4979-8937-4559fee41cf0"). InnerVolumeSpecName "kube-api-access-zlpls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:16:12.966114 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:12.966058    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31c08cbd-5315-4979-8937-4559fee41cf0" (UID: "31c08cbd-5315-4979-8937-4559fee41cf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:13.065270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.065223    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31c08cbd-5315-4979-8937-4559fee41cf0-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:13.065270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.065260    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31c08cbd-5315-4979-8937-4559fee41cf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:13.065270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.065274    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpls\" (UniqueName: \"kubernetes.io/projected/31c08cbd-5315-4979-8937-4559fee41cf0-kube-api-access-zlpls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:13.184723 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.184689    2575 generic.go:356] "Generic (PLEG): container finished" podID="31c08cbd-5315-4979-8937-4559fee41cf0" containerID="a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795" exitCode=0
Jul 07 15:16:13.185138 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.184737    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"
Jul 07 15:16:13.185138 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.184747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerDied","Data":"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"}
Jul 07 15:16:13.185138 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.184796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk" event={"ID":"31c08cbd-5315-4979-8937-4559fee41cf0","Type":"ContainerDied","Data":"57e0be39465346f4224accc7123f94bfb006c5b81ed7f0f04fdab3f7f3017945"}
Jul 07 15:16:13.185138 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.184821    2575 scope.go:117] "RemoveContainer" containerID="ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"
Jul 07 15:16:13.196111 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.196090    2575 scope.go:117] "RemoveContainer" containerID="a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"
Jul 07 15:16:13.204212 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.204180    2575 scope.go:117] "RemoveContainer" containerID="ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"
Jul 07 15:16:13.204549 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:16:13.204528    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac\": container with ID starting with ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac not found: ID does not exist" containerID="ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"
Jul 07 15:16:13.204624 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.204560    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac"} err="failed to get container status \"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac\": rpc error: code = NotFound desc = could not find container \"ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac\": container with ID starting with ae017795cc1cff8496e0cb82e9d045e2741b1d1a85eda37921e4556c398eacac not found: ID does not exist"
Jul 07 15:16:13.204624 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.204581    2575 scope.go:117] "RemoveContainer" containerID="a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"
Jul 07 15:16:13.204824 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:16:13.204808    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795\": container with ID starting with a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795 not found: ID does not exist" containerID="a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"
Jul 07 15:16:13.204866 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.204829    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795"} err="failed to get container status \"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795\": rpc error: code = NotFound desc = could not find container \"a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795\": container with ID starting with a6e1907407a842938a7613e975a538465c691a6dbab6e22893663f8a381ae795 not found: ID does not exist"
Jul 07 15:16:13.207630 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.207604    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:16:13.211157 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.211134    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c9c9-predictor-7d847f679-zghqk"]
Jul 07 15:16:13.412865 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:13.412824    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:14.919541 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:14.919507    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" path="/var/lib/kubelet/pods/31c08cbd-5315-4979-8937-4559fee41cf0/volumes"
Jul 07 15:16:17.184293 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:17.184262    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:16:17.184754 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:17.184717    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:18.412770 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:18.412731    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:23.412106 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:23.412061    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:23.412501 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:23.412178    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:16:27.185497 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:27.185457    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:28.413996 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:28.413957    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:33.412830 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:33.412790    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:37.184912 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:37.184867    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:38.412657 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:38.412613    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:16:39.017303 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.017273    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:16:39.074385 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.074350    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle\") pod \"86140778-5169-4d13-bad6-258d806b5763\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") "
Jul 07 15:16:39.074385 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.074382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls\") pod \"86140778-5169-4d13-bad6-258d806b5763\" (UID: \"86140778-5169-4d13-bad6-258d806b5763\") "
Jul 07 15:16:39.074737 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.074717    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "86140778-5169-4d13-bad6-258d806b5763" (UID: "86140778-5169-4d13-bad6-258d806b5763"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:39.076687 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.076655    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86140778-5169-4d13-bad6-258d806b5763" (UID: "86140778-5169-4d13-bad6-258d806b5763"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:39.175302 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.175210    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86140778-5169-4d13-bad6-258d806b5763-openshift-service-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:39.175302 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.175240    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86140778-5169-4d13-bad6-258d806b5763-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:39.286877 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.286841    2575 generic.go:356] "Generic (PLEG): container finished" podID="86140778-5169-4d13-bad6-258d806b5763" containerID="2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453" exitCode=0
Jul 07 15:16:39.287083 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.286899    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"
Jul 07 15:16:39.287083 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.286922    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" event={"ID":"86140778-5169-4d13-bad6-258d806b5763","Type":"ContainerDied","Data":"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"}
Jul 07 15:16:39.287083 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.287020    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q" event={"ID":"86140778-5169-4d13-bad6-258d806b5763","Type":"ContainerDied","Data":"d7ff89b71dc1338a6e62ab8538f98e33f64c519b1d78d4eda0d93b13a0eeb549"}
Jul 07 15:16:39.287083 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.287050    2575 scope.go:117] "RemoveContainer" containerID="2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"
Jul 07 15:16:39.295358 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.295338    2575 scope.go:117] "RemoveContainer" containerID="2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"
Jul 07 15:16:39.295684 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:16:39.295662    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453\": container with ID starting with 2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453 not found: ID does not exist" containerID="2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"
Jul 07 15:16:39.295751 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.295694    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453"} err="failed to get container status \"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453\": rpc error: code = NotFound desc = could not find container \"2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453\": container with ID starting with 2eaaf875f687fd9c3bb06fa1ff485d9e6fbfe1993d1c0a39bdd5ada524d1f453 not found: ID does not exist"
Jul 07 15:16:39.311652 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.311616    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:16:39.317474 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:39.317444    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3c9c9-7bd8bf8867-hjp6q"]
Jul 07 15:16:40.918813 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:40.918777    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86140778-5169-4d13-bad6-258d806b5763" path="/var/lib/kubelet/pods/86140778-5169-4d13-bad6-258d806b5763/volumes"
Jul 07 15:16:44.818958 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:44.818901    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:16:44.819395 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:44.819170    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container" containerID="cri-o://7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07" gracePeriod=30
Jul 07 15:16:44.819395 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:44.819230    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kube-rbac-proxy" containerID="cri-o://4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28" gracePeriod=30
Jul 07 15:16:45.103000 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.102890    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:16:45.103295 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103277    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container"
Jul 07 15:16:45.103348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103298    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container"
Jul 07 15:16:45.103348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103315    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9"
Jul 07 15:16:45.103348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103320    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9"
Jul 07 15:16:45.103348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103333    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kube-rbac-proxy"
Jul 07 15:16:45.103348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103339    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kube-rbac-proxy"
Jul 07 15:16:45.103510 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103387    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kube-rbac-proxy"
Jul 07 15:16:45.103510 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103394    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86140778-5169-4d13-bad6-258d806b5763" containerName="switch-graph-3c9c9"
Jul 07 15:16:45.103510 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.103400    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c08cbd-5315-4979-8937-4559fee41cf0" containerName="kserve-container"
Jul 07 15:16:45.105965 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.105945    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.109016 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.108990    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4605-kube-rbac-proxy-sar-config\""
Jul 07 15:16:45.109133 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.109039    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4605-predictor-serving-cert\""
Jul 07 15:16:45.125605 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.125574    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:16:45.224929 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.224889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.225133 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.224998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg9f\" (UniqueName: \"kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.225133 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.225061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.312841 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.312804    2575 generic.go:356] "Generic (PLEG): container finished" podID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerID="4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28" exitCode=2
Jul 07 15:16:45.313035 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.312882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerDied","Data":"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"}
Jul 07 15:16:45.326348 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.326311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.326472 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.326388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.326472 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.326417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg9f\" (UniqueName: \"kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.327226 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.327131    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.328909 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.328887    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.337519 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.337489    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg9f\" (UniqueName: \"kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f\") pod \"error-404-isvc-e4605-predictor-7f46859f89-sq7hs\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") " pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.416012 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.415901    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:45.555148 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:45.555116    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:16:45.558002 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:16:45.557975    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd241b8_ee2c_4fc0_9468_6328c4288214.slice/crio-20f7a500f0dd53a29b7aa5357f39b5477dbf63f543feb697cc0e0c2754b3abfd WatchSource:0}: Error finding container 20f7a500f0dd53a29b7aa5357f39b5477dbf63f543feb697cc0e0c2754b3abfd: Status 404 returned error can't find the container with id 20f7a500f0dd53a29b7aa5357f39b5477dbf63f543feb697cc0e0c2754b3abfd
Jul 07 15:16:46.318193 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:46.318151    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerStarted","Data":"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"}
Jul 07 15:16:46.318633 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:46.318222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerStarted","Data":"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"}
Jul 07 15:16:46.318633 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:46.318253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerStarted","Data":"20f7a500f0dd53a29b7aa5357f39b5477dbf63f543feb697cc0e0c2754b3abfd"}
Jul 07 15:16:46.318633 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:46.318390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:46.338526 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:46.338472    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podStartSLOduration=1.338455311 podStartE2EDuration="1.338455311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:16:46.33663624 +0000 UTC m=+1187.983175345" watchObservedRunningTime="2026-07-07 15:16:46.338455311 +0000 UTC m=+1187.984994413"
Jul 07 15:16:47.184887 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:47.184845    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 07 15:16:47.323385 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:47.323348    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:47.324676 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:47.324648    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:16:48.269889 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.269862    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:16:48.333558 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.333522    2575 generic.go:356] "Generic (PLEG): container finished" podID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerID="7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07" exitCode=0
Jul 07 15:16:48.333980 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.333588    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"
Jul 07 15:16:48.333980 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.333604    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerDied","Data":"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"}
Jul 07 15:16:48.333980 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.333706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv" event={"ID":"196ffcbf-7496-4f27-8599-d1b6d11fe543","Type":"ContainerDied","Data":"ff27d4b46c06c597ec216968eb4c24753c126bc1f41947144e9b2e65528a99a6"}
Jul 07 15:16:48.333980 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.333741    2575 scope.go:117] "RemoveContainer" containerID="4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"
Jul 07 15:16:48.334290 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.334266    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:16:48.341873 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.341852    2575 scope.go:117] "RemoveContainer" containerID="7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"
Jul 07 15:16:48.349598 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.349579    2575 scope.go:117] "RemoveContainer" containerID="4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"
Jul 07 15:16:48.349913 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:16:48.349889    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28\": container with ID starting with 4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28 not found: ID does not exist" containerID="4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"
Jul 07 15:16:48.349981 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.349926    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28"} err="failed to get container status \"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28\": rpc error: code = NotFound desc = could not find container \"4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28\": container with ID starting with 4ee08ede87d2d2c02938ddbf3e31857088ebfe90ef143775b32f89c123feda28 not found: ID does not exist"
Jul 07 15:16:48.350024 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.349982    2575 scope.go:117] "RemoveContainer" containerID="7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"
Jul 07 15:16:48.350256 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:16:48.350229    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07\": container with ID starting with 7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07 not found: ID does not exist" containerID="7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"
Jul 07 15:16:48.350352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.350259    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07"} err="failed to get container status \"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07\": rpc error: code = NotFound desc = could not find container \"7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07\": container with ID starting with 7fe383bf537fcf4cb6da15d6fd42a29c46402f4e4a8d69491f3e16b938db4e07 not found: ID does not exist"
Jul 07 15:16:48.352517 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.352497    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls\") pod \"196ffcbf-7496-4f27-8599-d1b6d11fe543\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") "
Jul 07 15:16:48.352611 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.352544    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrfkf\" (UniqueName: \"kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf\") pod \"196ffcbf-7496-4f27-8599-d1b6d11fe543\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") "
Jul 07 15:16:48.352611 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.352565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config\") pod \"196ffcbf-7496-4f27-8599-d1b6d11fe543\" (UID: \"196ffcbf-7496-4f27-8599-d1b6d11fe543\") "
Jul 07 15:16:48.352922 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.352899    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "196ffcbf-7496-4f27-8599-d1b6d11fe543" (UID: "196ffcbf-7496-4f27-8599-d1b6d11fe543"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:16:48.354612 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.354591    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "196ffcbf-7496-4f27-8599-d1b6d11fe543" (UID: "196ffcbf-7496-4f27-8599-d1b6d11fe543"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:16:48.354819 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.354803    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf" (OuterVolumeSpecName: "kube-api-access-xrfkf") pod "196ffcbf-7496-4f27-8599-d1b6d11fe543" (UID: "196ffcbf-7496-4f27-8599-d1b6d11fe543"). InnerVolumeSpecName "kube-api-access-xrfkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:16:48.453733 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.453687    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/196ffcbf-7496-4f27-8599-d1b6d11fe543-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:48.453733 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.453727    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrfkf\" (UniqueName: \"kubernetes.io/projected/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-api-access-xrfkf\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:48.453733 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.453742    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/196ffcbf-7496-4f27-8599-d1b6d11fe543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:16:48.655461 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.655427    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:16:48.661193 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.661168    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8996c-predictor-85d8b95c6c-hwmlv"]
Jul 07 15:16:48.919617 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:48.919538    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" path="/var/lib/kubelet/pods/196ffcbf-7496-4f27-8599-d1b6d11fe543/volumes"
Jul 07 15:16:53.338287 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:53.338260    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:16:53.338768 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:53.338743    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:16:57.185608 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:16:57.185580    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:17:03.339141 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:03.339097    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:17:13.339529 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:13.339483    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:17:19.513403 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.513362    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:17:19.513902 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.513684    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kube-rbac-proxy" containerID="cri-o://cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63" gracePeriod=30
Jul 07 15:17:19.513902 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.513675    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container" containerID="cri-o://9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333" gracePeriod=30
Jul 07 15:17:19.703606 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703568    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:17:19.703868 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703854    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container"
Jul 07 15:17:19.703868 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703868    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container"
Jul 07 15:17:19.704006 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703886    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kube-rbac-proxy"
Jul 07 15:17:19.704006 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703892    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kube-rbac-proxy"
Jul 07 15:17:19.704006 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703951    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kube-rbac-proxy"
Jul 07 15:17:19.704006 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.703962    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="196ffcbf-7496-4f27-8599-d1b6d11fe543" containerName="kserve-container"
Jul 07 15:17:19.706814 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.706796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.710115 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.710084    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5aeac-predictor-serving-cert\""
Jul 07 15:17:19.710115 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.710101    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5aeac-kube-rbac-proxy-sar-config\""
Jul 07 15:17:19.719426 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.719398    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:17:19.772664 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.772573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.772664 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.772615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbzq\" (UniqueName: \"kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.772846 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.772716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.873914 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.873873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.873914 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.873919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbzq\" (UniqueName: \"kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.874164 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.873987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.874586 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.874554    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.876588 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.876566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:19.882825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:19.882803    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbzq\" (UniqueName: \"kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq\") pod \"error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") " pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:20.016634 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.016596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:20.152971 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.152925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:17:20.155213 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:17:20.155185    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec4559f_825f_473b_a2b6_9ecf8bbe6429.slice/crio-c6d21311a80975a8d14d8cec215376800427c478e9397b85856965ecd7618669 WatchSource:0}: Error finding container c6d21311a80975a8d14d8cec215376800427c478e9397b85856965ecd7618669: Status 404 returned error can't find the container with id c6d21311a80975a8d14d8cec215376800427c478e9397b85856965ecd7618669
Jul 07 15:17:20.454464 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.454361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerStarted","Data":"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"}
Jul 07 15:17:20.454464 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.454422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerStarted","Data":"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"}
Jul 07 15:17:20.454687 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.454546    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:20.454687 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.454601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerStarted","Data":"c6d21311a80975a8d14d8cec215376800427c478e9397b85856965ecd7618669"}
Jul 07 15:17:20.454687 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.454639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:20.455753 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.455724    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:20.456153 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.456132    2575 generic.go:356] "Generic (PLEG): container finished" podID="cd396cf3-880b-4780-8cfa-c7589254de62" containerID="cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63" exitCode=2
Jul 07 15:17:20.456250 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.456173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerDied","Data":"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"}
Jul 07 15:17:20.475491 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:20.475436    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podStartSLOduration=1.475418071 podStartE2EDuration="1.475418071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:17:20.47414974 +0000 UTC m=+1222.120688842" watchObservedRunningTime="2026-07-07 15:17:20.475418071 +0000 UTC m=+1222.121957173"
Jul 07 15:17:21.463088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:21.463052    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:22.179457 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:22.179411    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 07 15:17:22.963969 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:22.963926    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:17:23.002407 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.002376    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls\") pod \"cd396cf3-880b-4780-8cfa-c7589254de62\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") "
Jul 07 15:17:23.002578 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.002421    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p9dz\" (UniqueName: \"kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz\") pod \"cd396cf3-880b-4780-8cfa-c7589254de62\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") "
Jul 07 15:17:23.002578 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.002474    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config\") pod \"cd396cf3-880b-4780-8cfa-c7589254de62\" (UID: \"cd396cf3-880b-4780-8cfa-c7589254de62\") "
Jul 07 15:17:23.002865 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.002835    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd396cf3-880b-4780-8cfa-c7589254de62" (UID: "cd396cf3-880b-4780-8cfa-c7589254de62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:17:23.004691 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.004667    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz" (OuterVolumeSpecName: "kube-api-access-8p9dz") pod "cd396cf3-880b-4780-8cfa-c7589254de62" (UID: "cd396cf3-880b-4780-8cfa-c7589254de62"). InnerVolumeSpecName "kube-api-access-8p9dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:17:23.004783 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.004688    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd396cf3-880b-4780-8cfa-c7589254de62" (UID: "cd396cf3-880b-4780-8cfa-c7589254de62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:17:23.103833 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.103747    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd396cf3-880b-4780-8cfa-c7589254de62-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:23.103833 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.103775    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9dz\" (UniqueName: \"kubernetes.io/projected/cd396cf3-880b-4780-8cfa-c7589254de62-kube-api-access-8p9dz\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:23.103833 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.103785    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd396cf3-880b-4780-8cfa-c7589254de62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:23.338841 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.338801    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 07 15:17:23.471047 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.470955    2575 generic.go:356] "Generic (PLEG): container finished" podID="cd396cf3-880b-4780-8cfa-c7589254de62" containerID="9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333" exitCode=0
Jul 07 15:17:23.471047 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.471013    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"
Jul 07 15:17:23.471047 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.471036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerDied","Data":"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"}
Jul 07 15:17:23.471261 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.471108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv" event={"ID":"cd396cf3-880b-4780-8cfa-c7589254de62","Type":"ContainerDied","Data":"4cc2382275067aaa46a0126c09fd6ffef8d4979504b4bac80706b3954b2a2737"}
Jul 07 15:17:23.471261 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.471147    2575 scope.go:117] "RemoveContainer" containerID="cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"
Jul 07 15:17:23.479165 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.479144    2575 scope.go:117] "RemoveContainer" containerID="9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"
Jul 07 15:17:23.486492 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.486472    2575 scope.go:117] "RemoveContainer" containerID="cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"
Jul 07 15:17:23.486782 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:17:23.486762    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63\": container with ID starting with cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63 not found: ID does not exist" containerID="cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"
Jul 07 15:17:23.486826 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.486793    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63"} err="failed to get container status \"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63\": rpc error: code = NotFound desc = could not find container \"cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63\": container with ID starting with cadc5c83e3e3d056e9cbde9bd1753aa1cea6b9de389b5e4e48f362e75ccdbf63 not found: ID does not exist"
Jul 07 15:17:23.486826 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.486812    2575 scope.go:117] "RemoveContainer" containerID="9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"
Jul 07 15:17:23.487075 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:17:23.487054    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333\": container with ID starting with 9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333 not found: ID does not exist" containerID="9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"
Jul 07 15:17:23.487132 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.487082    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333"} err="failed to get container status \"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333\": rpc error: code = NotFound desc = could not find container \"9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333\": container with ID starting with 9036af09f4990a8e5333fb792cbf996ee73ecc53892e178b77d18e04e59b7333 not found: ID does not exist"
Jul 07 15:17:23.493087 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.493060    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:17:23.500163 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:23.500136    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-542f9-predictor-5df9bc96cc-xnvdv"]
Jul 07 15:17:24.919310 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:24.919274    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" path="/var/lib/kubelet/pods/cd396cf3-880b-4780-8cfa-c7589254de62/volumes"
Jul 07 15:17:26.467512 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:26.467481    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:17:26.468135 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:26.468102    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:33.339537 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:33.339504    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:17:36.468738 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:36.468699    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:46.468876 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:46.468838    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:55.267538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267458    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267738    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kube-rbac-proxy"
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267749    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kube-rbac-proxy"
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267768    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container"
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267773    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container"
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267819    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kserve-container"
Jul 07 15:17:55.267895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.267826    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd396cf3-880b-4780-8cfa-c7589254de62" containerName="kube-rbac-proxy"
Jul 07 15:17:55.271859 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.271793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.275040 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.275012    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9c06-predictor-serving-cert\""
Jul 07 15:17:55.275264 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.275246    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9c06-kube-rbac-proxy-sar-config\""
Jul 07 15:17:55.281895 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.281856    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:17:55.337391 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.337354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:17:55.337626 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.337580    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container" containerID="cri-o://0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d" gracePeriod=30
Jul 07 15:17:55.337718 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.337607    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kube-rbac-proxy" containerID="cri-o://94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee" gracePeriod=30
Jul 07 15:17:55.352910 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.352866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2kvm\" (UniqueName: \"kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.353056 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.352970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.353056 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.353015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.454140 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.454094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2kvm\" (UniqueName: \"kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.454295 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.454191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.454295 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.454248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.454882 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.454853    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.456723 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.456699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.464774 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.464749    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2kvm\" (UniqueName: \"kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm\") pod \"error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") " pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.585206 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.583347    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:55.588244 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.588210    2575 generic.go:356] "Generic (PLEG): container finished" podID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerID="94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee" exitCode=2
Jul 07 15:17:55.588378 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.588285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerDied","Data":"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"}
Jul 07 15:17:55.722991 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:55.722810    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:17:55.725648 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:17:55.725617    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da5da07_7a4e_4229_9fbf_e91d5e9def8b.slice/crio-f5bb337cc2939c9e933977a1ca35b6142824a5c6b9758053b2ce1f9effd6e82e WatchSource:0}: Error finding container f5bb337cc2939c9e933977a1ca35b6142824a5c6b9758053b2ce1f9effd6e82e: Status 404 returned error can't find the container with id f5bb337cc2939c9e933977a1ca35b6142824a5c6b9758053b2ce1f9effd6e82e
Jul 07 15:17:56.468062 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.468025    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:17:56.595089 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.595043    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerStarted","Data":"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"}
Jul 07 15:17:56.595270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.595124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerStarted","Data":"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"}
Jul 07 15:17:56.595270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.595156    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerStarted","Data":"f5bb337cc2939c9e933977a1ca35b6142824a5c6b9758053b2ce1f9effd6e82e"}
Jul 07 15:17:56.595270 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.595193    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:56.616032 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:56.615979    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podStartSLOduration=1.61595835 podStartE2EDuration="1.61595835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:17:56.61475999 +0000 UTC m=+1258.261299095" watchObservedRunningTime="2026-07-07 15:17:56.61595835 +0000 UTC m=+1258.262497447"
Jul 07 15:17:57.599481 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:57.599450    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:17:57.600683 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:57.600654    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:17:58.335404 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:58.335360    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 07 15:17:58.603506 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:58.603416    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:17:59.291553 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.291526    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:17:59.387699 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.387659    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config\") pod \"5bd241b8-ee2c-4fc0-9468-6328c4288214\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") "
Jul 07 15:17:59.387912 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.387766    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls\") pod \"5bd241b8-ee2c-4fc0-9468-6328c4288214\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") "
Jul 07 15:17:59.387912 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.387868    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcg9f\" (UniqueName: \"kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f\") pod \"5bd241b8-ee2c-4fc0-9468-6328c4288214\" (UID: \"5bd241b8-ee2c-4fc0-9468-6328c4288214\") "
Jul 07 15:17:59.388169 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.388143    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bd241b8-ee2c-4fc0-9468-6328c4288214" (UID: "5bd241b8-ee2c-4fc0-9468-6328c4288214"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:17:59.390139 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.390105    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bd241b8-ee2c-4fc0-9468-6328c4288214" (UID: "5bd241b8-ee2c-4fc0-9468-6328c4288214"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:17:59.392867 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.390221    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f" (OuterVolumeSpecName: "kube-api-access-qcg9f") pod "5bd241b8-ee2c-4fc0-9468-6328c4288214" (UID: "5bd241b8-ee2c-4fc0-9468-6328c4288214"). InnerVolumeSpecName "kube-api-access-qcg9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:17:59.488872 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.488770    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:59.488872 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.488820    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd241b8-ee2c-4fc0-9468-6328c4288214-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:59.488872 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.488840    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcg9f\" (UniqueName: \"kubernetes.io/projected/5bd241b8-ee2c-4fc0-9468-6328c4288214-kube-api-access-qcg9f\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:17:59.608671 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.608635    2575 generic.go:356] "Generic (PLEG): container finished" podID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerID="0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d" exitCode=0
Jul 07 15:17:59.609093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.608700    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"
Jul 07 15:17:59.609093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.608725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerDied","Data":"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"}
Jul 07 15:17:59.609093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.608804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs" event={"ID":"5bd241b8-ee2c-4fc0-9468-6328c4288214","Type":"ContainerDied","Data":"20f7a500f0dd53a29b7aa5357f39b5477dbf63f543feb697cc0e0c2754b3abfd"}
Jul 07 15:17:59.609093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.608830    2575 scope.go:117] "RemoveContainer" containerID="94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"
Jul 07 15:17:59.617698 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.617679    2575 scope.go:117] "RemoveContainer" containerID="0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"
Jul 07 15:17:59.625494 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.625466    2575 scope.go:117] "RemoveContainer" containerID="94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"
Jul 07 15:17:59.625752 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:17:59.625732    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee\": container with ID starting with 94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee not found: ID does not exist" containerID="94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"
Jul 07 15:17:59.625821 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.625761    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee"} err="failed to get container status \"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee\": rpc error: code = NotFound desc = could not find container \"94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee\": container with ID starting with 94a13c2301287441574ff2149ee7b8cf40f78c942ff7a77aaa484ab9ec0a82ee not found: ID does not exist"
Jul 07 15:17:59.625821 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.625780    2575 scope.go:117] "RemoveContainer" containerID="0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"
Jul 07 15:17:59.626167 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:17:59.626146    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d\": container with ID starting with 0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d not found: ID does not exist" containerID="0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"
Jul 07 15:17:59.626206 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.626175    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d"} err="failed to get container status \"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d\": rpc error: code = NotFound desc = could not find container \"0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d\": container with ID starting with 0b084a3711a51f25d08ee66a8dda9c5745058c90e3bf5f1c2e3a14674918146d not found: ID does not exist"
Jul 07 15:17:59.644498 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.644455    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:17:59.657051 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:17:59.657020    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4605-predictor-7f46859f89-sq7hs"]
Jul 07 15:18:00.920091 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:00.920061    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" path="/var/lib/kubelet/pods/5bd241b8-ee2c-4fc0-9468-6328c4288214/volumes"
Jul 07 15:18:03.607881 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:03.607854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:18:03.608362 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:03.608336    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:18:06.468661 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:06.468629    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:18:13.608745 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:13.608704    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:18:19.547628 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547590    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547889    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kube-rbac-proxy"
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547901    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kube-rbac-proxy"
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547911    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container"
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547916    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container"
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.547996    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kube-rbac-proxy"
Jul 07 15:18:19.548090 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.548007    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd241b8-ee2c-4fc0-9468-6328c4288214" containerName="kserve-container"
Jul 07 15:18:19.552309 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.552292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.557462 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.557437    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5aeac-serving-cert\""
Jul 07 15:18:19.557596 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.557551    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5aeac-kube-rbac-proxy-sar-config\""
Jul 07 15:18:19.569111 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.569082    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:18:19.647768 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.647733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.648010 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.647789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.748529 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.748486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.748732 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.748543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.749177 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.749157    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.751122 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.751098    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls\") pod \"ensemble-graph-5aeac-8558df8fd8-mz2x2\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") " pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.862108 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.862017    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:19.999258 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:19.999233    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:18:20.001626 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:18:20.001595    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ac236_3402_4144_b4ec_902df600ec17.slice/crio-6c111549024391ea3b5c5d123645de615a50c4eb678661618cc43d29182468b4 WatchSource:0}: Error finding container 6c111549024391ea3b5c5d123645de615a50c4eb678661618cc43d29182468b4: Status 404 returned error can't find the container with id 6c111549024391ea3b5c5d123645de615a50c4eb678661618cc43d29182468b4
Jul 07 15:18:20.691088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:20.691052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" event={"ID":"1c8ac236-3402-4144-b4ec-902df600ec17","Type":"ContainerStarted","Data":"b4f6b21c3ec4a8ef1b0e695f4d77a3ea5efbcfaaea8fc7cf0ed9bc32d74fdf59"}
Jul 07 15:18:20.691518 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:20.691107    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" event={"ID":"1c8ac236-3402-4144-b4ec-902df600ec17","Type":"ContainerStarted","Data":"6c111549024391ea3b5c5d123645de615a50c4eb678661618cc43d29182468b4"}
Jul 07 15:18:20.691518 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:20.691148    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:20.714782 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:20.714727    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podStartSLOduration=1.714711589 podStartE2EDuration="1.714711589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:18:20.712654722 +0000 UTC m=+1282.359193824" watchObservedRunningTime="2026-07-07 15:18:20.714711589 +0000 UTC m=+1282.361250693"
Jul 07 15:18:23.609174 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:23.609138    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:18:26.700717 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:26.700685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:18:33.609243 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:33.609203    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 15:18:43.609108 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:18:43.609077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:26:34.245888 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.245854    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:26:34.248283 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.246065    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" containerID="cri-o://b4f6b21c3ec4a8ef1b0e695f4d77a3ea5efbcfaaea8fc7cf0ed9bc32d74fdf59" gracePeriod=30
Jul 07 15:26:34.781772 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.781735    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:26:34.784987 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.784970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.787312 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.787282    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-761e8-predictor-serving-cert\""
Jul 07 15:26:34.787453 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.787426    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-761e8-kube-rbac-proxy-sar-config\""
Jul 07 15:26:34.800224 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.800196    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:26:34.864310 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.864279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnx8\" (UniqueName: \"kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.864484 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.864335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.864484 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.864376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.867320 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.867290    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:26:34.867546 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.867520    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" containerID="cri-o://2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687" gracePeriod=30
Jul 07 15:26:34.867600 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.867562    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kube-rbac-proxy" containerID="cri-o://f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da" gracePeriod=30
Jul 07 15:26:34.965551 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.965508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.965711 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.965556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.965781 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.965755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnx8\" (UniqueName: \"kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.966433 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.966357    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.968278 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.968257    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:34.983166 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:34.983129    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnx8\" (UniqueName: \"kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8\") pod \"error-404-isvc-761e8-predictor-7869d59c55-4qcxf\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") " pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:35.095151 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.095052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:35.242266 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.242243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:26:35.244701 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:26:35.244669    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ff71cc_091f_473a_bac2_cc9977ab8fc5.slice/crio-61725c3a5308016f40d3c35067dd20b7527bc2f32f2b995092230f2efd45264c WatchSource:0}: Error finding container 61725c3a5308016f40d3c35067dd20b7527bc2f32f2b995092230f2efd45264c: Status 404 returned error can't find the container with id 61725c3a5308016f40d3c35067dd20b7527bc2f32f2b995092230f2efd45264c
Jul 07 15:26:35.493377 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.493324    2575 generic.go:356] "Generic (PLEG): container finished" podID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerID="f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da" exitCode=2
Jul 07 15:26:35.493924 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.493407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerDied","Data":"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"}
Jul 07 15:26:35.495923 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.495877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerStarted","Data":"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"}
Jul 07 15:26:35.496072 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.495979    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerStarted","Data":"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"}
Jul 07 15:26:35.496072 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.496014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerStarted","Data":"61725c3a5308016f40d3c35067dd20b7527bc2f32f2b995092230f2efd45264c"}
Jul 07 15:26:35.496172 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.496066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:35.517148 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:35.517083    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podStartSLOduration=1.5170628499999999 podStartE2EDuration="1.51706285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:26:35.51699946 +0000 UTC m=+1777.163538562" watchObservedRunningTime="2026-07-07 15:26:35.51706285 +0000 UTC m=+1777.163601959"
Jul 07 15:26:36.464091 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:36.464043    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 15:26:36.468366 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:36.468340    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 15:26:36.500072 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:36.500038    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:36.501430 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:36.501404    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:26:36.699493 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:36.699457    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:26:37.503635 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:37.503589    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:26:38.426962 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.426920    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:26:38.494529 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.494423    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls\") pod \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") "
Jul 07 15:26:38.494713 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.494541    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbzq\" (UniqueName: \"kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq\") pod \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") "
Jul 07 15:26:38.494713 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.494568    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config\") pod \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\" (UID: \"0ec4559f-825f-473b-a2b6-9ecf8bbe6429\") "
Jul 07 15:26:38.495088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.495053    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ec4559f-825f-473b-a2b6-9ecf8bbe6429" (UID: "0ec4559f-825f-473b-a2b6-9ecf8bbe6429"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:26:38.496790 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.496761    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ec4559f-825f-473b-a2b6-9ecf8bbe6429" (UID: "0ec4559f-825f-473b-a2b6-9ecf8bbe6429"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:26:38.496790 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.496770    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq" (OuterVolumeSpecName: "kube-api-access-lzbzq") pod "0ec4559f-825f-473b-a2b6-9ecf8bbe6429" (UID: "0ec4559f-825f-473b-a2b6-9ecf8bbe6429"). InnerVolumeSpecName "kube-api-access-lzbzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:26:38.510184 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.510154    2575 generic.go:356] "Generic (PLEG): container finished" podID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerID="2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687" exitCode=0
Jul 07 15:26:38.510625 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.510259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerDied","Data":"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"}
Jul 07 15:26:38.510625 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.510277    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"
Jul 07 15:26:38.510625 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.510331    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx" event={"ID":"0ec4559f-825f-473b-a2b6-9ecf8bbe6429","Type":"ContainerDied","Data":"c6d21311a80975a8d14d8cec215376800427c478e9397b85856965ecd7618669"}
Jul 07 15:26:38.510625 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.510370    2575 scope.go:117] "RemoveContainer" containerID="f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"
Jul 07 15:26:38.519086 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.519067    2575 scope.go:117] "RemoveContainer" containerID="2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"
Jul 07 15:26:38.528494 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.528473    2575 scope.go:117] "RemoveContainer" containerID="f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"
Jul 07 15:26:38.528813 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:26:38.528791    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da\": container with ID starting with f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da not found: ID does not exist" containerID="f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"
Jul 07 15:26:38.528866 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.528824    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da"} err="failed to get container status \"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da\": rpc error: code = NotFound desc = could not find container \"f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da\": container with ID starting with f6830980d69c3bde9ae51310b8591a25c07b2a65d6e76dc89c4deb9abb2f31da not found: ID does not exist"
Jul 07 15:26:38.528866 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.528843    2575 scope.go:117] "RemoveContainer" containerID="2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"
Jul 07 15:26:38.529147 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:26:38.529126    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687\": container with ID starting with 2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687 not found: ID does not exist" containerID="2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"
Jul 07 15:26:38.529275 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.529150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687"} err="failed to get container status \"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687\": rpc error: code = NotFound desc = could not find container \"2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687\": container with ID starting with 2907c5ed2ec9d41f0abaf5b0270bc37e9c447d3fb7f7da7e7fc80854e989a687 not found: ID does not exist"
Jul 07 15:26:38.538626 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.538597    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:26:38.552897 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.552867    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5aeac-predictor-59dfb478c9-8pbmx"]
Jul 07 15:26:38.595878 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.595842    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbzq\" (UniqueName: \"kubernetes.io/projected/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-api-access-lzbzq\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:26:38.595878 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.595875    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:26:38.596095 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.595887    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec4559f-825f-473b-a2b6-9ecf8bbe6429-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:26:38.920538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:38.920504    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" path="/var/lib/kubelet/pods/0ec4559f-825f-473b-a2b6-9ecf8bbe6429/volumes"
Jul 07 15:26:41.699399 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:41.699361    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:26:42.508258 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:42.508227    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:26:42.508707 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:42.508671    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:26:46.698839 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:46.698796    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:26:46.699283 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:46.698916    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:26:51.699396 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:51.699315    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:26:52.509163 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:52.509123    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:26:56.699028 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:26:56.698986    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:27:01.698954 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:01.698905    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:27:02.508780 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:02.508745    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:27:04.610480 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:04.610445    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c8ac236-3402-4144-b4ec-902df600ec17" containerID="b4f6b21c3ec4a8ef1b0e695f4d77a3ea5efbcfaaea8fc7cf0ed9bc32d74fdf59" exitCode=0
Jul 07 15:27:04.610856 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:04.610505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" event={"ID":"1c8ac236-3402-4144-b4ec-902df600ec17","Type":"ContainerDied","Data":"b4f6b21c3ec4a8ef1b0e695f4d77a3ea5efbcfaaea8fc7cf0ed9bc32d74fdf59"}
Jul 07 15:27:04.899778 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:04.899746    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:27:05.019093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.019057    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle\") pod \"1c8ac236-3402-4144-b4ec-902df600ec17\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") "
Jul 07 15:27:05.019093 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.019109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls\") pod \"1c8ac236-3402-4144-b4ec-902df600ec17\" (UID: \"1c8ac236-3402-4144-b4ec-902df600ec17\") "
Jul 07 15:27:05.019464 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.019433    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1c8ac236-3402-4144-b4ec-902df600ec17" (UID: "1c8ac236-3402-4144-b4ec-902df600ec17"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:27:05.021366 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.021345    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c8ac236-3402-4144-b4ec-902df600ec17" (UID: "1c8ac236-3402-4144-b4ec-902df600ec17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:27:05.120322 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.120284    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c8ac236-3402-4144-b4ec-902df600ec17-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:05.120322 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.120318    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c8ac236-3402-4144-b4ec-902df600ec17-openshift-service-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:05.614885 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.614854    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"
Jul 07 15:27:05.615489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.614853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2" event={"ID":"1c8ac236-3402-4144-b4ec-902df600ec17","Type":"ContainerDied","Data":"6c111549024391ea3b5c5d123645de615a50c4eb678661618cc43d29182468b4"}
Jul 07 15:27:05.615489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.615005    2575 scope.go:117] "RemoveContainer" containerID="b4f6b21c3ec4a8ef1b0e695f4d77a3ea5efbcfaaea8fc7cf0ed9bc32d74fdf59"
Jul 07 15:27:05.645918 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.645886    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:27:05.648819 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:05.648788    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5aeac-8558df8fd8-mz2x2"]
Jul 07 15:27:06.919104 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:06.919069    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" path="/var/lib/kubelet/pods/1c8ac236-3402-4144-b4ec-902df600ec17/volumes"
Jul 07 15:27:10.012810 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.012774    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:27:10.013223 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.013034    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container" containerID="cri-o://a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e" gracePeriod=30
Jul 07 15:27:10.013223 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.013061    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kube-rbac-proxy" containerID="cri-o://c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59" gracePeriod=30
Jul 07 15:27:10.362419 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362336    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:27:10.362629 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362617    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac"
Jul 07 15:27:10.362674 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362630    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac"
Jul 07 15:27:10.362674 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362642    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container"
Jul 07 15:27:10.362674 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362648    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container"
Jul 07 15:27:10.362674 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362658    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kube-rbac-proxy"
Jul 07 15:27:10.362674 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362663    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kube-rbac-proxy"
Jul 07 15:27:10.362811 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362720    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8ac236-3402-4144-b4ec-902df600ec17" containerName="ensemble-graph-5aeac"
Jul 07 15:27:10.362811 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362733    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kserve-container"
Jul 07 15:27:10.362811 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.362740    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec4559f-825f-473b-a2b6-9ecf8bbe6429" containerName="kube-rbac-proxy"
Jul 07 15:27:10.365801 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.365783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.370168 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.370143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e9b-kube-rbac-proxy-sar-config\""
Jul 07 15:27:10.370326 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.370211    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e9b-predictor-serving-cert\""
Jul 07 15:27:10.389254 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.389219    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:27:10.562708 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.562668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zh2q\" (UniqueName: \"kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.562887 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.562728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.562887 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.562814    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.635806 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.635715    2575 generic.go:356] "Generic (PLEG): container finished" podID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerID="c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59" exitCode=2
Jul 07 15:27:10.635806 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.635786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerDied","Data":"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"}
Jul 07 15:27:10.664180 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.664144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.664280 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.664194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zh2q\" (UniqueName: \"kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.664280 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.664248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.664896 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.664867    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.666883 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.666861    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.675307 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.675281    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zh2q\" (UniqueName: \"kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q\") pod \"error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.675438 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.675413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:10.835807 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:10.835776    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:27:10.841851 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:27:10.841820    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49efee85_7499_48bf_b795_523fc057f59d.slice/crio-b7ae7f4fac6253bba1da5ab4c5c8968eed5a989a2c4f6ef3a59de5ab4d022e36 WatchSource:0}: Error finding container b7ae7f4fac6253bba1da5ab4c5c8968eed5a989a2c4f6ef3a59de5ab4d022e36: Status 404 returned error can't find the container with id b7ae7f4fac6253bba1da5ab4c5c8968eed5a989a2c4f6ef3a59de5ab4d022e36
Jul 07 15:27:11.641083 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:11.641042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerStarted","Data":"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"}
Jul 07 15:27:11.641485 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:11.641120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:11.641485 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:11.641153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerStarted","Data":"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"}
Jul 07 15:27:11.641485 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:11.641184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerStarted","Data":"b7ae7f4fac6253bba1da5ab4c5c8968eed5a989a2c4f6ef3a59de5ab4d022e36"}
Jul 07 15:27:11.672491 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:11.672443    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podStartSLOduration=1.672427915 podStartE2EDuration="1.672427915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:27:11.671656398 +0000 UTC m=+1813.318195502" watchObservedRunningTime="2026-07-07 15:27:11.672427915 +0000 UTC m=+1813.318967019"
Jul 07 15:27:12.508807 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:12.508771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 15:27:12.645335 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:12.645302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:12.646722 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:12.646693    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:13.567729 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.567699    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:27:13.584556 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.584528    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls\") pod \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") "
Jul 07 15:27:13.584704 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.584578    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2kvm\" (UniqueName: \"kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm\") pod \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") "
Jul 07 15:27:13.584704 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.584623    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config\") pod \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\" (UID: \"9da5da07-7a4e-4229-9fbf-e91d5e9def8b\") "
Jul 07 15:27:13.585007 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.584970    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9da5da07-7a4e-4229-9fbf-e91d5e9def8b" (UID: "9da5da07-7a4e-4229-9fbf-e91d5e9def8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:27:13.586825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.586801    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9da5da07-7a4e-4229-9fbf-e91d5e9def8b" (UID: "9da5da07-7a4e-4229-9fbf-e91d5e9def8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:27:13.586825 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.586809    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm" (OuterVolumeSpecName: "kube-api-access-g2kvm") pod "9da5da07-7a4e-4229-9fbf-e91d5e9def8b" (UID: "9da5da07-7a4e-4229-9fbf-e91d5e9def8b"). InnerVolumeSpecName "kube-api-access-g2kvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:27:13.650868 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.650774    2575 generic.go:356] "Generic (PLEG): container finished" podID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerID="a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e" exitCode=0
Jul 07 15:27:13.650868 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.650826    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"
Jul 07 15:27:13.651347 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.650868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerDied","Data":"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"}
Jul 07 15:27:13.651347 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.650929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk" event={"ID":"9da5da07-7a4e-4229-9fbf-e91d5e9def8b","Type":"ContainerDied","Data":"f5bb337cc2939c9e933977a1ca35b6142824a5c6b9758053b2ce1f9effd6e82e"}
Jul 07 15:27:13.651347 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.650963    2575 scope.go:117] "RemoveContainer" containerID="c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"
Jul 07 15:27:13.651501 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.651405    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:13.659718 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.659672    2575 scope.go:117] "RemoveContainer" containerID="a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"
Jul 07 15:27:13.667560 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.667534    2575 scope.go:117] "RemoveContainer" containerID="c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"
Jul 07 15:27:13.667855 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:13.667835    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59\": container with ID starting with c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59 not found: ID does not exist" containerID="c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"
Jul 07 15:27:13.667905 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.667866    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59"} err="failed to get container status \"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59\": rpc error: code = NotFound desc = could not find container \"c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59\": container with ID starting with c1a29882c53789eb222da4586877ffb92f09c3ecea17220ccefad4133982be59 not found: ID does not exist"
Jul 07 15:27:13.667905 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.667884    2575 scope.go:117] "RemoveContainer" containerID="a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"
Jul 07 15:27:13.668179 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:13.668158    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e\": container with ID starting with a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e not found: ID does not exist" containerID="a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"
Jul 07 15:27:13.668227 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.668186    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e"} err="failed to get container status \"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e\": rpc error: code = NotFound desc = could not find container \"a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e\": container with ID starting with a10d1e5b03ff3b51697def49afecff67fa1122e86d02e11610e026144ba31a9e not found: ID does not exist"
Jul 07 15:27:13.679364 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.679336    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:27:13.685406 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.685383    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:13.685473 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.685411    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2kvm\" (UniqueName: \"kubernetes.io/projected/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-api-access-g2kvm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:13.685473 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.685422    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9da5da07-7a4e-4229-9fbf-e91d5e9def8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:13.694144 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:13.694116    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9c06-predictor-77c555c7dc-n9tbk"]
Jul 07 15:27:14.919452 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:14.919420    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" path="/var/lib/kubelet/pods/9da5da07-7a4e-4229-9fbf-e91d5e9def8b/volumes"
Jul 07 15:27:18.656669 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:18.656641    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:27:18.657241 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:18.657216    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:22.509847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:22.509818    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:27:28.657777 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:28.657741    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:38.658190 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:38.658145    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:44.727369 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:44.727330    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:27:44.727812 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:44.727597    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container" containerID="cri-o://74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4" gracePeriod=30
Jul 07 15:27:44.727812 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:44.727649    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kube-rbac-proxy" containerID="cri-o://6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac" gracePeriod=30
Jul 07 15:27:45.021547 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021509    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:27:45.021821 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021806    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kube-rbac-proxy"
Jul 07 15:27:45.021881 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021822    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kube-rbac-proxy"
Jul 07 15:27:45.021881 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021839    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container"
Jul 07 15:27:45.021881 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021844    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container"
Jul 07 15:27:45.022071 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021915    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kube-rbac-proxy"
Jul 07 15:27:45.022071 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.021927    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da5da07-7a4e-4229-9fbf-e91d5e9def8b" containerName="kserve-container"
Jul 07 15:27:45.024874 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.024852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.029239 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.029211    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-73e96-predictor-serving-cert\""
Jul 07 15:27:45.029416 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.029229    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-73e96-kube-rbac-proxy-sar-config\""
Jul 07 15:27:45.041273 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.041240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:27:45.130034 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.129982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.130233 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.130047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lr9\" (UniqueName: \"kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.130233 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.130082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.230580 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.230532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.230580 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.230581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lr9\" (UniqueName: \"kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.230849 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.230600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.230849 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:45.230711    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-73e96-predictor-serving-cert: secret "error-404-isvc-73e96-predictor-serving-cert" not found
Jul 07 15:27:45.230849 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:45.230784    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls podName:c2eb94fe-213a-402e-942d-5ce48933167b nodeName:}" failed. No retries permitted until 2026-07-07 15:27:45.730764452 +0000 UTC m=+1847.377303535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls") pod "error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" (UID: "c2eb94fe-213a-402e-942d-5ce48933167b") : secret "error-404-isvc-73e96-predictor-serving-cert" not found
Jul 07 15:27:45.231300 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.231273    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.242634 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.242608    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lr9\" (UniqueName: \"kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.735991 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.735926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.738564 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.738540    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") pod \"error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") " pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:45.772647 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.772607    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerID="6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac" exitCode=2
Jul 07 15:27:45.772813 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.772658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerDied","Data":"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"}
Jul 07 15:27:45.936499 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:45.936457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:46.077745 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.077721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:27:46.080396 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:27:46.080359    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2eb94fe_213a_402e_942d_5ce48933167b.slice/crio-71a4831f07ccaf8364b08a776be484ea8ce73a501fdaf92042eb78836a4ca0c9 WatchSource:0}: Error finding container 71a4831f07ccaf8364b08a776be484ea8ce73a501fdaf92042eb78836a4ca0c9: Status 404 returned error can't find the container with id 71a4831f07ccaf8364b08a776be484ea8ce73a501fdaf92042eb78836a4ca0c9
Jul 07 15:27:46.779144 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.779092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerStarted","Data":"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"}
Jul 07 15:27:46.779618 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.779156    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:46.779618 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.779176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerStarted","Data":"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"}
Jul 07 15:27:46.779618 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.779195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerStarted","Data":"71a4831f07ccaf8364b08a776be484ea8ce73a501fdaf92042eb78836a4ca0c9"}
Jul 07 15:27:46.779618 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.779213    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:46.780741 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.780715    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:27:46.801870 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:46.801803    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podStartSLOduration=2.801779486 podStartE2EDuration="2.801779486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:27:46.801175978 +0000 UTC m=+1848.447715083" watchObservedRunningTime="2026-07-07 15:27:46.801779486 +0000 UTC m=+1848.448318589"
Jul 07 15:27:47.504335 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:47.504292    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 07 15:27:47.782838 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:47.782746    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:27:48.181638 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.181598    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:27:48.257569 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.257531    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config\") pod \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") "
Jul 07 15:27:48.257750 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.257595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls\") pod \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") "
Jul 07 15:27:48.257750 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.257712    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlnx8\" (UniqueName: \"kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8\") pod \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\" (UID: \"b1ff71cc-091f-473a-bac2-cc9977ab8fc5\") "
Jul 07 15:27:48.257986 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.257957    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1ff71cc-091f-473a-bac2-cc9977ab8fc5" (UID: "b1ff71cc-091f-473a-bac2-cc9977ab8fc5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:27:48.259917 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.259888    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1ff71cc-091f-473a-bac2-cc9977ab8fc5" (UID: "b1ff71cc-091f-473a-bac2-cc9977ab8fc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:27:48.259917 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.259907    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8" (OuterVolumeSpecName: "kube-api-access-tlnx8") pod "b1ff71cc-091f-473a-bac2-cc9977ab8fc5" (UID: "b1ff71cc-091f-473a-bac2-cc9977ab8fc5"). InnerVolumeSpecName "kube-api-access-tlnx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:27:48.359307 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.359204    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:48.359307 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.359249    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:48.359307 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.359260    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnx8\" (UniqueName: \"kubernetes.io/projected/b1ff71cc-091f-473a-bac2-cc9977ab8fc5-kube-api-access-tlnx8\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:27:48.657696 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.657602    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 07 15:27:48.788009 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.787963    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerID="74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4" exitCode=0
Jul 07 15:27:48.788489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.788033    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"
Jul 07 15:27:48.788489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.788066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerDied","Data":"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"}
Jul 07 15:27:48.788489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.788185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf" event={"ID":"b1ff71cc-091f-473a-bac2-cc9977ab8fc5","Type":"ContainerDied","Data":"61725c3a5308016f40d3c35067dd20b7527bc2f32f2b995092230f2efd45264c"}
Jul 07 15:27:48.788489 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.788224    2575 scope.go:117] "RemoveContainer" containerID="6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"
Jul 07 15:27:48.797556 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.797533    2575 scope.go:117] "RemoveContainer" containerID="74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"
Jul 07 15:27:48.805579 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.805559    2575 scope.go:117] "RemoveContainer" containerID="6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"
Jul 07 15:27:48.805917 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:48.805897    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac\": container with ID starting with 6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac not found: ID does not exist" containerID="6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"
Jul 07 15:27:48.805993 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.805929    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac"} err="failed to get container status \"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac\": rpc error: code = NotFound desc = could not find container \"6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac\": container with ID starting with 6c8dfcdcf4b28e99e45f485586006a0bdabcfb0a328d01bdaa0173616498c4ac not found: ID does not exist"
Jul 07 15:27:48.805993 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.805969    2575 scope.go:117] "RemoveContainer" containerID="74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"
Jul 07 15:27:48.806220 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:27:48.806202    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4\": container with ID starting with 74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4 not found: ID does not exist" containerID="74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"
Jul 07 15:27:48.806263 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.806225    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4"} err="failed to get container status \"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4\": rpc error: code = NotFound desc = could not find container \"74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4\": container with ID starting with 74b37a3c0f564a48ee3d53a3bb425b046f0e467e1d9d657ec89a0c9c00df09c4 not found: ID does not exist"
Jul 07 15:27:48.814363 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.814335    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:27:48.817590 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.817564    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-761e8-predictor-7869d59c55-4qcxf"]
Jul 07 15:27:48.919585 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:48.919489    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" path="/var/lib/kubelet/pods/b1ff71cc-091f-473a-bac2-cc9977ab8fc5/volumes"
Jul 07 15:27:52.787137 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:52.787099    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:27:52.787669 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:52.787641    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:27:58.658011 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:27:58.657965    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:28:02.787600 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:02.787559    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.179253    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.179842    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.179861    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.179889    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kube-rbac-proxy"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.179898    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kube-rbac-proxy"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.180077    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kserve-container"
Jul 07 15:28:10.181352 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.180090    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ff71cc-091f-473a-bac2-cc9977ab8fc5" containerName="kube-rbac-proxy"
Jul 07 15:28:10.183473 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.183444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.189630 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.189601    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c3e9b-kube-rbac-proxy-sar-config\""
Jul 07 15:28:10.190142 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.189601    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c3e9b-serving-cert\""
Jul 07 15:28:10.217910 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.217873    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:28:10.338498 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.338454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.338685 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.338508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.439864 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.439769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.439864 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.439819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.440515 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.440494    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.442427 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.442396    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls\") pod \"switch-graph-c3e9b-7f8b6f4d7d-5twgt\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") " pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.494154 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.494116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.634444 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.634374    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:28:10.636630 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:28:10.636604    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03391917_8310_4234_b9f4_2d3d2a425067.slice/crio-51a88e483d8eb544aaa81b74a0fbad71bb9e7c1cd701b5122929db551b3d7e4d WatchSource:0}: Error finding container 51a88e483d8eb544aaa81b74a0fbad71bb9e7c1cd701b5122929db551b3d7e4d: Status 404 returned error can't find the container with id 51a88e483d8eb544aaa81b74a0fbad71bb9e7c1cd701b5122929db551b3d7e4d
Jul 07 15:28:10.871869 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.871837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" event={"ID":"03391917-8310-4234-b9f4-2d3d2a425067","Type":"ContainerStarted","Data":"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"}
Jul 07 15:28:10.872089 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.871891    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" event={"ID":"03391917-8310-4234-b9f4-2d3d2a425067","Type":"ContainerStarted","Data":"51a88e483d8eb544aaa81b74a0fbad71bb9e7c1cd701b5122929db551b3d7e4d"}
Jul 07 15:28:10.872089 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.871963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:10.896899 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:10.896827    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podStartSLOduration=0.896803464 podStartE2EDuration="896.803464ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:28:10.896255165 +0000 UTC m=+1872.542794269" watchObservedRunningTime="2026-07-07 15:28:10.896803464 +0000 UTC m=+1872.543342569"
Jul 07 15:28:12.787596 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:12.787557    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:28:16.880604 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:16.880574    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:28:22.788618 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:22.788528    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:28:32.788126 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:28:32.788089    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:36:59.740173 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:36:59.740136    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:36:59.742633 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:36:59.740370    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" containerID="cri-o://4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e" gracePeriod=30
Jul 07 15:36:59.742633 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:36:59.740421    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kube-rbac-proxy" containerID="cri-o://11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c" gracePeriod=30
Jul 07 15:37:00.814709 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:00.814676    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2eb94fe-213a-402e-942d-5ce48933167b" containerID="11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c" exitCode=2
Jul 07 15:37:00.815121 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:00.814753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerDied","Data":"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"}
Jul 07 15:37:02.784026 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:02.783981    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 15:37:02.788370 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:02.788331    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 15:37:03.092880 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.092854    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:37:03.178118 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.178084    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config\") pod \"c2eb94fe-213a-402e-942d-5ce48933167b\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") "
Jul 07 15:37:03.178337 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.178139    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") pod \"c2eb94fe-213a-402e-942d-5ce48933167b\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") "
Jul 07 15:37:03.178337 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.178264    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4lr9\" (UniqueName: \"kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9\") pod \"c2eb94fe-213a-402e-942d-5ce48933167b\" (UID: \"c2eb94fe-213a-402e-942d-5ce48933167b\") "
Jul 07 15:37:03.178471 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.178451    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2eb94fe-213a-402e-942d-5ce48933167b" (UID: "c2eb94fe-213a-402e-942d-5ce48933167b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:37:03.180393 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.180365    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2eb94fe-213a-402e-942d-5ce48933167b" (UID: "c2eb94fe-213a-402e-942d-5ce48933167b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:37:03.180393 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.180380    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9" (OuterVolumeSpecName: "kube-api-access-f4lr9") pod "c2eb94fe-213a-402e-942d-5ce48933167b" (UID: "c2eb94fe-213a-402e-942d-5ce48933167b"). InnerVolumeSpecName "kube-api-access-f4lr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:37:03.279400 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.279365    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2eb94fe-213a-402e-942d-5ce48933167b-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:37:03.279400 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.279395    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lr9\" (UniqueName: \"kubernetes.io/projected/c2eb94fe-213a-402e-942d-5ce48933167b-kube-api-access-f4lr9\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:37:03.279400 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.279405    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2eb94fe-213a-402e-942d-5ce48933167b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:37:03.826946 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.826911    2575 generic.go:356] "Generic (PLEG): container finished" podID="c2eb94fe-213a-402e-942d-5ce48933167b" containerID="4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e" exitCode=0
Jul 07 15:37:03.827334 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.826980    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"
Jul 07 15:37:03.827334 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.826980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerDied","Data":"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"}
Jul 07 15:37:03.827334 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.827104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8" event={"ID":"c2eb94fe-213a-402e-942d-5ce48933167b","Type":"ContainerDied","Data":"71a4831f07ccaf8364b08a776be484ea8ce73a501fdaf92042eb78836a4ca0c9"}
Jul 07 15:37:03.827334 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.827133    2575 scope.go:117] "RemoveContainer" containerID="11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"
Jul 07 15:37:03.835798 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.835778    2575 scope.go:117] "RemoveContainer" containerID="4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"
Jul 07 15:37:03.843681 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.843662    2575 scope.go:117] "RemoveContainer" containerID="11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"
Jul 07 15:37:03.844044 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:37:03.844018    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c\": container with ID starting with 11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c not found: ID does not exist" containerID="11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"
Jul 07 15:37:03.844140 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.844055    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c"} err="failed to get container status \"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c\": rpc error: code = NotFound desc = could not find container \"11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c\": container with ID starting with 11d1aea38c8beb9c98df7688506d5be39e8eccbe26062de586ba947de0c3513c not found: ID does not exist"
Jul 07 15:37:03.844140 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.844071    2575 scope.go:117] "RemoveContainer" containerID="4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"
Jul 07 15:37:03.844345 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:37:03.844328    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e\": container with ID starting with 4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e not found: ID does not exist" containerID="4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"
Jul 07 15:37:03.844381 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.844352    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e"} err="failed to get container status \"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e\": rpc error: code = NotFound desc = could not find container \"4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e\": container with ID starting with 4366942e16b71532ad5957e897ad864b63840f1dd439914a27300928c7e0e63e not found: ID does not exist"
Jul 07 15:37:03.848745 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.848719    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:37:03.853149 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:03.853128    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73e96-predictor-78d9dc85b9-bh7l8"]
Jul 07 15:37:04.918671 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:37:04.918638    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" path="/var/lib/kubelet/pods/c2eb94fe-213a-402e-942d-5ce48933167b/volumes"
Jul 07 15:44:29.375997 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:29.375964    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:44:29.378468 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:29.376214    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" containerID="cri-o://86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5" gracePeriod=30
Jul 07 15:44:29.627641 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:29.627553    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:44:29.627875 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:29.627810    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container" containerID="cri-o://f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d" gracePeriod=30
Jul 07 15:44:29.627984 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:29.627853    2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kube-rbac-proxy" containerID="cri-o://ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc" gracePeriod=30
Jul 07 15:44:30.425888 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:30.425853    2575 generic.go:356] "Generic (PLEG): container finished" podID="49efee85-7499-48bf-b795-523fc057f59d" containerID="ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc" exitCode=2
Jul 07 15:44:30.426292 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:30.425897    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerDied","Data":"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"}
Jul 07 15:44:31.879241 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:31.879201    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:32.785384 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.785356    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:44:32.886502 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.886404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls\") pod \"49efee85-7499-48bf-b795-523fc057f59d\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") "
Jul 07 15:44:32.886502 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.886448    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zh2q\" (UniqueName: \"kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q\") pod \"49efee85-7499-48bf-b795-523fc057f59d\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") "
Jul 07 15:44:32.886502 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.886476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config\") pod \"49efee85-7499-48bf-b795-523fc057f59d\" (UID: \"49efee85-7499-48bf-b795-523fc057f59d\") "
Jul 07 15:44:32.886968 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.886838    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49efee85-7499-48bf-b795-523fc057f59d" (UID: "49efee85-7499-48bf-b795-523fc057f59d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:44:32.888676 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.888654    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q" (OuterVolumeSpecName: "kube-api-access-9zh2q") pod "49efee85-7499-48bf-b795-523fc057f59d" (UID: "49efee85-7499-48bf-b795-523fc057f59d"). InnerVolumeSpecName "kube-api-access-9zh2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:44:32.888727 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.888707    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49efee85-7499-48bf-b795-523fc057f59d" (UID: "49efee85-7499-48bf-b795-523fc057f59d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:44:32.987965 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.987897    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49efee85-7499-48bf-b795-523fc057f59d-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:44:32.987965 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.987956    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zh2q\" (UniqueName: \"kubernetes.io/projected/49efee85-7499-48bf-b795-523fc057f59d-kube-api-access-9zh2q\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:44:32.987965 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:32.987969    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49efee85-7499-48bf-b795-523fc057f59d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:44:33.437437 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.437400    2575 generic.go:356] "Generic (PLEG): container finished" podID="49efee85-7499-48bf-b795-523fc057f59d" containerID="f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d" exitCode=0
Jul 07 15:44:33.437613 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.437458    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"
Jul 07 15:44:33.437613 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.437482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerDied","Data":"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"}
Jul 07 15:44:33.437613 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.437553    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x" event={"ID":"49efee85-7499-48bf-b795-523fc057f59d","Type":"ContainerDied","Data":"b7ae7f4fac6253bba1da5ab4c5c8968eed5a989a2c4f6ef3a59de5ab4d022e36"}
Jul 07 15:44:33.437613 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.437579    2575 scope.go:117] "RemoveContainer" containerID="ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"
Jul 07 15:44:33.445750 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.445731    2575 scope.go:117] "RemoveContainer" containerID="f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"
Jul 07 15:44:33.453397 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.453365    2575 scope.go:117] "RemoveContainer" containerID="ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"
Jul 07 15:44:33.453745 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:44:33.453707    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc\": container with ID starting with ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc not found: ID does not exist" containerID="ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"
Jul 07 15:44:33.453815 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.453759    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc"} err="failed to get container status \"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc\": rpc error: code = NotFound desc = could not find container \"ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc\": container with ID starting with ab5f4421b39ef9574f7134d7f049bb63370e92eef12bd374d4f8748aec49ddbc not found: ID does not exist"
Jul 07 15:44:33.453815 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.453785    2575 scope.go:117] "RemoveContainer" containerID="f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"
Jul 07 15:44:33.454141 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:44:33.454119    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d\": container with ID starting with f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d not found: ID does not exist" containerID="f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"
Jul 07 15:44:33.454209 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.454150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d"} err="failed to get container status \"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d\": rpc error: code = NotFound desc = could not find container \"f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d\": container with ID starting with f657423598f8c6292285ed52ab490fc9f9cf7222d883ae7017ca4ad8ea946e4d not found: ID does not exist"
Jul 07 15:44:33.454853 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.454834    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:44:33.459885 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:33.459862    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e9b-predictor-7578b96d6f-9zh9x"]
Jul 07 15:44:34.919066 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:34.919034    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49efee85-7499-48bf-b795-523fc057f59d" path="/var/lib/kubelet/pods/49efee85-7499-48bf-b795-523fc057f59d/volumes"
Jul 07 15:44:36.879652 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:36.879615    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:41.880434 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:41.880388    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:41.880887 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:41.880501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:44:45.859712 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:45.859685    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:46.712667 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:46.712622    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:46.879716 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:46.879672    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:47.562131 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:47.562097    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:48.346039 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:48.346008    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:49.171568 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:49.171523    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:49.986767 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:49.986741    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:50.790477 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:50.790450    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:51.603920 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:51.603882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:51.880225 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:51.880136    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:52.398632 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:52.398602    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:53.206997 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:53.206962    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:54.036601 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:54.036563    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:54.888141 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:54.888114    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:55.729564 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:55.729529    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:56.603241 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:56.603212    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-c3e9b-7f8b6f4d7d-5twgt_03391917-8310-4234-b9f4-2d3d2a425067/switch-graph-c3e9b/0.log"
Jul 07 15:44:56.879902 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:56.879817    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:44:59.518509 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.518480    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:44:59.537121 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.537084    2575 generic.go:356] "Generic (PLEG): container finished" podID="03391917-8310-4234-b9f4-2d3d2a425067" containerID="86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5" exitCode=0
Jul 07 15:44:59.537299 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.537142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" event={"ID":"03391917-8310-4234-b9f4-2d3d2a425067","Type":"ContainerDied","Data":"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"}
Jul 07 15:44:59.537299 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.537178    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"
Jul 07 15:44:59.537299 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.537197    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt" event={"ID":"03391917-8310-4234-b9f4-2d3d2a425067","Type":"ContainerDied","Data":"51a88e483d8eb544aaa81b74a0fbad71bb9e7c1cd701b5122929db551b3d7e4d"}
Jul 07 15:44:59.537299 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.537229    2575 scope.go:117] "RemoveContainer" containerID="86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"
Jul 07 15:44:59.545490 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.545469    2575 scope.go:117] "RemoveContainer" containerID="86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"
Jul 07 15:44:59.545783 ip-10-0-128-215 kubenswrapper[2575]: E0707 15:44:59.545764    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5\": container with ID starting with 86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5 not found: ID does not exist" containerID="86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"
Jul 07 15:44:59.545847 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.545794    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5"} err="failed to get container status \"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5\": rpc error: code = NotFound desc = could not find container \"86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5\": container with ID starting with 86ef0f778e18d9b3e64aa1988f4178f6555b6898728ce127663b4e8759d70ce5 not found: ID does not exist"
Jul 07 15:44:59.591264 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.591226    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle\") pod \"03391917-8310-4234-b9f4-2d3d2a425067\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") "
Jul 07 15:44:59.591264 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.591272    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls\") pod \"03391917-8310-4234-b9f4-2d3d2a425067\" (UID: \"03391917-8310-4234-b9f4-2d3d2a425067\") "
Jul 07 15:44:59.591614 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.591588    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "03391917-8310-4234-b9f4-2d3d2a425067" (UID: "03391917-8310-4234-b9f4-2d3d2a425067"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:44:59.593611 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.593586    2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03391917-8310-4234-b9f4-2d3d2a425067" (UID: "03391917-8310-4234-b9f4-2d3d2a425067"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:44:59.692067 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.691967    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03391917-8310-4234-b9f4-2d3d2a425067-openshift-service-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:44:59.692067 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.692005    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03391917-8310-4234-b9f4-2d3d2a425067-proxy-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 07 15:44:59.858403 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.858366    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:44:59.861788 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:44:59.861760    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c3e9b-7f8b6f4d7d-5twgt"]
Jul 07 15:45:00.919440 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:00.919400    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03391917-8310-4234-b9f4-2d3d2a425067" path="/var/lib/kubelet/pods/03391917-8310-4234-b9f4-2d3d2a425067/volumes"
Jul 07 15:45:04.811675 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:04.811644    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lt87x_2aafa307-bcb5-42de-953b-596ec07d8845/global-pull-secret-syncer/0.log"
Jul 07 15:45:05.014302 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:05.014270    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zrkcb_58e1c6b8-f561-4076-9157-299ed47c3827/konnectivity-agent/0.log"
Jul 07 15:45:05.036571 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:05.036544    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-215.ec2.internal_04dfa3a43b9c6e6928c1a69c4428406e/haproxy/0.log"
Jul 07 15:45:09.387983 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.387952    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-lqxfb_f62f4c33-fcea-49b9-931b-da0952304209/monitoring-plugin/0.log"
Jul 07 15:45:09.427390 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.427351    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-597sm_aeab4723-c9f2-42e8-b3bf-fe0dea37a35b/node-exporter/0.log"
Jul 07 15:45:09.456197 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.456167    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-597sm_aeab4723-c9f2-42e8-b3bf-fe0dea37a35b/kube-rbac-proxy/0.log"
Jul 07 15:45:09.489138 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.489107    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-597sm_aeab4723-c9f2-42e8-b3bf-fe0dea37a35b/init-textfile/0.log"
Jul 07 15:45:09.950346 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.950309    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2fm26_60a05fda-c0dc-495a-8c3c-f4f8e836c43b/prometheus-operator/0.log"
Jul 07 15:45:09.972601 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:09.972575    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2fm26_60a05fda-c0dc-495a-8c3c-f4f8e836c43b/kube-rbac-proxy/0.log"
Jul 07 15:45:12.931628 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931591    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"]
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931867    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931878    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931889    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931895    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931905    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931910    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931919    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931928    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931958    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.931964    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.932013    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.932023    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.932029    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="49efee85-7499-48bf-b795-523fc057f59d" containerName="kube-rbac-proxy"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.932034    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2eb94fe-213a-402e-942d-5ce48933167b" containerName="kserve-container"
Jul 07 15:45:12.932041 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.932039    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="03391917-8310-4234-b9f4-2d3d2a425067" containerName="switch-graph-c3e9b"
Jul 07 15:45:12.934828 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.934808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:12.937178 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.937161    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xx6g\"/\"kube-root-ca.crt\""
Jul 07 15:45:12.937928 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.937915    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4xx6g\"/\"default-dockercfg-s4f8t\""
Jul 07 15:45:12.938005 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.937953    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xx6g\"/\"openshift-service-ca.crt\""
Jul 07 15:45:12.944116 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:12.944085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"]
Jul 07 15:45:13.093154 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.093118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-podres\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.093373 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.093181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-proc\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.093373 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.093206    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-lib-modules\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.093373 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.093241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76qj\" (UniqueName: \"kubernetes.io/projected/30948384-e939-4154-af75-db33f8b6bbf2-kube-api-access-f76qj\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.093373 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.093270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-sys\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194229 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76qj\" (UniqueName: \"kubernetes.io/projected/30948384-e939-4154-af75-db33f8b6bbf2-kube-api-access-f76qj\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194229 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-sys\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194229 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-podres\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194229 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-proc\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-lib-modules\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-sys\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-proc\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-lib-modules\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.194538 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.194388    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30948384-e939-4154-af75-db33f8b6bbf2-podres\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.202416 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.202384    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76qj\" (UniqueName: \"kubernetes.io/projected/30948384-e939-4154-af75-db33f8b6bbf2-kube-api-access-f76qj\") pod \"perf-node-gather-daemonset-w58dk\" (UID: \"30948384-e939-4154-af75-db33f8b6bbf2\") " pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.245404 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.245359    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.378198 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.378150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"]
Jul 07 15:45:13.381086 ip-10-0-128-215 kubenswrapper[2575]: W0707 15:45:13.381044    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30948384_e939_4154_af75_db33f8b6bbf2.slice/crio-a03ea6118f8d6d29502d5f5f1ccf76a171cccdf498c374393b92d86e55d64407 WatchSource:0}: Error finding container a03ea6118f8d6d29502d5f5f1ccf76a171cccdf498c374393b92d86e55d64407: Status 404 returned error can't find the container with id a03ea6118f8d6d29502d5f5f1ccf76a171cccdf498c374393b92d86e55d64407
Jul 07 15:45:13.588994 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.588954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk" event={"ID":"30948384-e939-4154-af75-db33f8b6bbf2","Type":"ContainerStarted","Data":"14c9938a1a138c579f776bf90fc687184a397d12c58ae855eb39168700fe5e45"}
Jul 07 15:45:13.589179 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.589013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk" event={"ID":"30948384-e939-4154-af75-db33f8b6bbf2","Type":"ContainerStarted","Data":"a03ea6118f8d6d29502d5f5f1ccf76a171cccdf498c374393b92d86e55d64407"}
Jul 07 15:45:13.589179 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.589082    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:13.606531 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.606477    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk" podStartSLOduration=1.6064587559999999 podStartE2EDuration="1.606458756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:45:13.605188038 +0000 UTC m=+2895.251727146" watchObservedRunningTime="2026-07-07 15:45:13.606458756 +0000 UTC m=+2895.252997914"
Jul 07 15:45:13.984821 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:13.984799    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wc5dm_ab6f2819-988c-484b-87db-2a7b9a7f5b58/dns/0.log"
Jul 07 15:45:14.009654 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:14.009625    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wc5dm_ab6f2819-988c-484b-87db-2a7b9a7f5b58/kube-rbac-proxy/0.log"
Jul 07 15:45:14.145079 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:14.145047    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2rvd7_74cf5a19-c3f7-414c-963f-ac9a91e6f235/dns-node-resolver/0.log"
Jul 07 15:45:14.689719 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:14.689676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cc7df4966-ljnd4_55a9ebce-7148-4feb-93a7-cb0158b5754e/registry/0.log"
Jul 07 15:45:14.768225 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:14.768196    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szttf_740fa349-e5f2-4369-935e-8ee9ff32fb74/node-ca/0.log"
Jul 07 15:45:16.014706 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:16.014676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9hs6q_0720eaae-7119-4509-93a1-5660aa611213/serve-healthcheck-canary/0.log"
Jul 07 15:45:16.542085 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:16.542056    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wjbk_117767b5-9775-4096-b6d2-ccc50508bcae/kube-rbac-proxy/0.log"
Jul 07 15:45:16.567815 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:16.567784    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wjbk_117767b5-9775-4096-b6d2-ccc50508bcae/exporter/0.log"
Jul 07 15:45:16.593753 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:16.593724    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6wjbk_117767b5-9775-4096-b6d2-ccc50508bcae/extractor/0.log"
Jul 07 15:45:18.963365 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:18.963341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-qnmkk_a03ce0de-8345-4147-8df5-0b660f383fd8/manager/0.log"
Jul 07 15:45:18.988336 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:18.988308    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-fc8zs_38eb894d-c153-4c96-8081-e7834231378a/server/0.log"
Jul 07 15:45:19.602469 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:19.602442    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4xx6g/perf-node-gather-daemonset-w58dk"
Jul 07 15:45:24.740753 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:24.740722    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6fgxt_a5395966-5e5b-4dca-a539-1281a2cc96df/kube-multus/0.log"
Jul 07 15:45:24.951172 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:24.951144    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/kube-multus-additional-cni-plugins/0.log"
Jul 07 15:45:24.976258 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:24.976231    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/egress-router-binary-copy/0.log"
Jul 07 15:45:25.002088 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.002012    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/cni-plugins/0.log"
Jul 07 15:45:25.025117 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.025092    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/bond-cni-plugin/0.log"
Jul 07 15:45:25.049965 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.049921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/routeoverride-cni/0.log"
Jul 07 15:45:25.072306 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.072278    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/whereabouts-cni-bincopy/0.log"
Jul 07 15:45:25.099718 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.099687    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzczk_1aea3089-4245-4500-80c3-3fc0151b6c96/whereabouts-cni/0.log"
Jul 07 15:45:25.326402 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.326368    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gqtvk_cd05c438-977e-42f1-ac3a-10605f8f2d41/network-metrics-daemon/0.log"
Jul 07 15:45:25.349827 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:25.349792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gqtvk_cd05c438-977e-42f1-ac3a-10605f8f2d41/kube-rbac-proxy/0.log"
Jul 07 15:45:26.219804 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.219766    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/ovn-controller/0.log"
Jul 07 15:45:26.241325 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.241289    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/ovn-acl-logging/0.log"
Jul 07 15:45:26.265426 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.265402    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/kube-rbac-proxy-node/0.log"
Jul 07 15:45:26.289822 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.289794    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 15:45:26.312717 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.312694    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/northd/0.log"
Jul 07 15:45:26.336569 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.336540    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/nbdb/0.log"
Jul 07 15:45:26.366276 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.366251    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/sbdb/0.log"
Jul 07 15:45:26.411805 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:26.411777    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ftfml_70524e5b-7ba0-4696-b5c1-76288ad7c029/ovnkube-controller/0.log"
Jul 07 15:45:28.001321 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:28.001295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rbkcv_1f878f43-2b0f-4463-968a-34052c7d9eef/network-check-target-container/0.log"
Jul 07 15:45:29.010462 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:29.010433    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2q8zf_42e7abb5-296e-4142-9ed6-0859b5613eb3/iptables-alerter/0.log"
Jul 07 15:45:29.772746 ip-10-0-128-215 kubenswrapper[2575]: I0707 15:45:29.772720    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-qfn9v_b6497f60-17b2-4e9c-bd83-17c0f13f3956/tuned/0.log"