Jul 07 02:15:51.884494 ip-10-0-137-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 02:15:51.884504 ip-10-0-137-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 02:15:51.884511 ip-10-0-137-109 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 02:15:51.884984 ip-10-0-137-109 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 02:16:03.053669 ip-10-0-137-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 02:16:03.053685 ip-10-0-137-109 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9f6bcb3d270c477788dc6115ead4f39c -- Jul 07 02:17:55.111192 ip-10-0-137-109 systemd[1]: Starting Kubernetes Kubelet... Jul 07 02:17:55.535520 ip-10-0-137-109 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:55.535520 ip-10-0-137-109 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 02:17:55.535520 ip-10-0-137-109 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:55.535520 ip-10-0-137-109 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 02:17:55.535520 ip-10-0-137-109 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 02:17:55.537693 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.537515 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.544966 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.544997 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545006 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545010 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545014 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545018 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545022 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.545008 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545027 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545031 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545035 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545039 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545048 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545052 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545057 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545060 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545064 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545068 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545072 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545075 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545079 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545082 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545086 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545090 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545097 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545101 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545105 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545110 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.545498 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545113 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545117 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545120 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545125 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545128 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545132 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545135 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545139 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545143 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545151 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545155 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545159 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545165 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545169 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545174 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545179 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545184 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545189 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545195 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.546354 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545201 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545206 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545215 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545220 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545224 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545229 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545239 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545247 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545252 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545257 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545262 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545267 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545271 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545276 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545286 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545290 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545295 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545301 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545305 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545310 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.547219 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545314 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545318 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545322 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545327 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545332 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545336 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545342 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545352 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545357 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545361 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545366 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545376 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545381 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545386 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545391 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545396 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545405 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545411 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545420 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.548079 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545426 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545431 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545437 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545442 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545448 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545452 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545458 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545465 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545470 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545475 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545479 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545484 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545492 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.545498 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547597 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547614 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547627 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547633 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547638 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547643 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.548855 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547649 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547654 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547660 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547664 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547669 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547674 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547680 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547687 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547699 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547705 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547712 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547717 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547722 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547726 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547731 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547736 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547741 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547746 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547751 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547756 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.549462 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547760 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547770 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547802 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547808 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547813 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547818 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547822 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547826 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547831 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547836 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547841 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547845 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547855 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547859 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547864 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547868 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547873 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547877 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547883 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.550121 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547888 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547893 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547897 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547903 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547907 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547912 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547920 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547925 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547930 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547934 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547939 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547943 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547948 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547952 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547956 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547960 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547966 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547971 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547975 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547984 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.550616 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547988 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.547993 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548001 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548005 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548010 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548016 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548022 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548026 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548031 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548035 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548039 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548048 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548054 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548059 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548064 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548068 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548073 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548079 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548084 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.551176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548088 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548093 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548096 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548101 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548110 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548116 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548121 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548126 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548131 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548135 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548140 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548145 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548150 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548155 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.548159 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549730 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549747 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549750 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549753 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549756 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.551647 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549759 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549762 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549765 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549767 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549772 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549775 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549796 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549800 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549802 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549805 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549808 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549810 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549814 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549816 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549819 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549821 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549824 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549829 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549834 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549837 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.552176 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549840 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549843 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549846 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549850 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549853 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549856 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549859 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549862 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549865 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549868 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549871 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549874 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549877 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549880 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549883 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549886 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549888 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549891 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549894 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549896 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.552662 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549899 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549903 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549907 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549910 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549914 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549917 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549920 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549922 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549925 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549928 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549931 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549933 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549936 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549938 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549941 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549944 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549947 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549949 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549951 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.553302 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549954 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549957 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549960 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549963 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549968 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549970 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549973 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549975 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549977 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549980 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549983 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549985 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549987 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549990 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549992 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549995 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.549997 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550000 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550003 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.553800 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550005 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550008 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550014 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550017 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550019 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550022 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550026 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550030 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550033 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550035 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550048 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550050 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550053 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550056 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550059 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550062 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550148 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550157 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550165 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550170 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550175 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 02:17:55.554282 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550179 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550184 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550188 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550192 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550195 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550198 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550202 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550205 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550208 2564 flags.go:64] FLAG: --cgroup-root="" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550211 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550214 2564 flags.go:64] FLAG: --client-ca-file="" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550218 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550221 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550229 2564 flags.go:64] FLAG: --cluster-domain="" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550232 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550236 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550239 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550242 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550246 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550249 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550252 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550256 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550259 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550262 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 02:17:55.554833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550265 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550269 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550272 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550277 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550279 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550282 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550285 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550288 2564 flags.go:64] FLAG: --enable-server="true" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550291 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550296 2564 flags.go:64] FLAG: --event-burst="100" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550299 2564 flags.go:64] FLAG: --event-qps="50" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550302 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550305 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550308 2564 flags.go:64] FLAG: --eviction-hard="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550312 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550315 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550317 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550321 2564 flags.go:64] FLAG: --eviction-soft="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550324 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550327 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550330 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550335 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550338 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550340 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550343 2564 flags.go:64] FLAG: --feature-gates="" Jul 07 02:17:55.555399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550347 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550350 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550353 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550356 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550359 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550362 2564 flags.go:64] FLAG: --help="false" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550365 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550368 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550371 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550374 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550377 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550380 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550383 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550385 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550388 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550391 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550393 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550396 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550399 2564 flags.go:64] FLAG: --kube-reserved="" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550402 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550405 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550408 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550410 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550413 2564 flags.go:64] FLAG: --lock-file="" Jul 07 02:17:55.556130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550415 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550418 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550420 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550427 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550432 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550435 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550438 2564 flags.go:64] FLAG: --logging-format="text" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550440 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550444 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550446 2564 flags.go:64] FLAG: --manifest-url="" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550449 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550455 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550457 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550462 2564 flags.go:64] FLAG: --max-pods="110" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550465 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550469 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550472 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550474 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550477 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550480 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550483 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550492 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550495 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550498 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 02:17:55.556734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550501 2564 flags.go:64] FLAG: --pod-cidr="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550504 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550509 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550512 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550516 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550518 2564 flags.go:64] FLAG: --port="10250" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550521 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550525 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09df73ce9fe919a23" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550528 2564 flags.go:64] FLAG: --qos-reserved="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550531 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550534 2564 flags.go:64] FLAG: --register-node="true" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550537 2564 flags.go:64] FLAG: --register-with-taints="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550542 2564 flags.go:64] FLAG: --registry-burst="10" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550547 2564 flags.go:64] FLAG: --registry-qps="5" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550550 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550553 2564 flags.go:64] FLAG: --reserved-memory="" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550557 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550560 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550563 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550566 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550568 2564 flags.go:64] FLAG: --runonce="false" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550571 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550574 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550578 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550580 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 02:17:55.557369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550583 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550586 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550589 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550592 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550595 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550598 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550601 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550604 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550607 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550609 2564 flags.go:64] FLAG: --system-cgroups="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550612 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550618 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550621 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550624 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550628 2564 flags.go:64] FLAG: --tls-min-version="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550631 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550634 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550637 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550640 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550644 2564 flags.go:64] FLAG: --v="2" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550649 2564 flags.go:64] FLAG: --version="false" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550654 2564 flags.go:64] FLAG: --vmodule="" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550658 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.550661 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550815 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.558026 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550820 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550823 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550827 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550829 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550835 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550837 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550840 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550843 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550845 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550848 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550850 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550853 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550856 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550859 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550862 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550864 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550868 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550871 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550873 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550875 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.558721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550878 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550880 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550883 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550885 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550888 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550890 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550893 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550895 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550898 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550901 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550904 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550908 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550912 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550915 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550918 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550921 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550926 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550929 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550931 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.559255 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550935 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550937 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550940 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550942 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550945 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550948 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550950 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550954 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550957 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550960 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550962 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550965 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550968 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550970 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550973 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550976 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550978 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550981 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550983 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550986 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.559721 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550988 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550992 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550995 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.550999 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551002 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551005 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551007 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551009 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551012 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551016 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551019 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551021 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551023 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551026 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551029 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551031 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551034 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551038 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551041 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551044 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.560274 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551046 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551049 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551052 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551055 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551057 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551059 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551062 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551064 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551067 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551070 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551073 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551075 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551078 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551080 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551083 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551085 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551088 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551091 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.560766 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.551093 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.551994 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.560188 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.560215 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.560239 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.560245 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560313 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560319 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560322 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560325 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560328 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560331 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560334 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560336 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.561224 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560339 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560342 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560344 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560347 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560350 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560352 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560354 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560358 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560360 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560363 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560365 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560368 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560371 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560373 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560377 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560381 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560384 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560387 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560390 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.561573 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560392 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560394 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560399 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560402 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560405 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560407 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560409 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560412 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560415 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560417 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560420 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560422 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560425 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560427 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560430 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560432 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560435 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560437 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560440 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560442 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.562070 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560445 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560447 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560450 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560453 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560456 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560458 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560460 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560463 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560466 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560468 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560471 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560473 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560476 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560479 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560485 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560488 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560491 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560493 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560496 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.562565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560499 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560501 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560504 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560506 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560509 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560511 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560513 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560515 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560518 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560520 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560523 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560525 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560528 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560530 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560533 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560535 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560538 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560540 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560544 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560547 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.563057 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560550 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560553 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560555 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560558 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560561 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560563 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560566 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560569 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560572 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560575 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560578 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560581 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560583 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.560588 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560718 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 02:17:55.563558 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560723 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560726 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560728 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560731 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560733 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560736 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560738 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560741 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560743 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560746 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560748 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560750 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560754 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560757 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560760 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560763 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560766 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560769 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 02:17:55.563967 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560777 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560797 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560802 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560805 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560807 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560810 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560814 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560817 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560820 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560823 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560825 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560828 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560830 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560833 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560835 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560838 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560841 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560844 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560846 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 02:17:55.564429 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560849 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560852 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560854 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560856 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560859 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560861 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560864 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560867 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560869 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560871 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560874 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560877 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560879 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560882 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560884 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560887 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560889 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560892 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560895 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560897 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 02:17:55.564954 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560900 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560903 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560906 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560909 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560911 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560914 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560916 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560919 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560921 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560923 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560926 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560928 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560931 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560933 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560935 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560938 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560940 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560943 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560946 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560949 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 02:17:55.565446 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560951 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560954 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560956 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560959 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560961 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560963 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560966 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560968 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560970 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560973 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560975 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560978 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560981 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560984 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560987 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560990 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560992 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560996 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.560999 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 02:17:55.565966 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:55.561002 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.561007 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.561175 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.563559 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.565159 2564 server.go:1019] "Starting client certificate rotation" Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.565276 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 02:17:55.566448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.565307 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 02:17:55.567728 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.567709 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 02:17:55.569774 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.569752 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 02:17:55.586854 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.586809 2564 log.go:25] "Validated CRI v1 runtime API" Jul 07 02:17:55.590616 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.590589 2564 log.go:25] "Validated CRI v1 image API" Jul 07 02:17:55.591691 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.591667 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 02:17:55.596342 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.596308 2564 fs.go:135] Filesystem UUIDs: map[5c1ed7b7-7210-41be-8efa-0439faeb1081:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ca84e2fd-88d1-4f3f-bdd2-c27de2f6ff26:/dev/nvme0n1p3] Jul 07 02:17:55.596342 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.596338 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 02:17:55.598801 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.598755 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 02:17:55.602847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.602655 2564 manager.go:216] Machine: {Timestamp:2026-07-07 02:17:55.600300035 +0000 UTC m=+0.357849072 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099431 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df499a35a7c5c62571ddeb56d69d0 SystemUUID:ec2df499-a35a-7c5c-6257-1ddeb56d69d0 BootID:9f6bcb3d-270c-4777-88dc-6115ead4f39c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c0:60:01:d8:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c0:60:01:d8:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:8d:44:f5:8e:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 02:17:55.602847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.602822 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 02:17:55.603061 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.602982 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 02:17:55.606108 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.606062 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 02:17:55.606261 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.606112 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 02:17:55.606923 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.606907 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 02:17:55.606959 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.606926 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 02:17:55.606959 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.606936 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 02:17:55.607547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.607531 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 02:17:55.608861 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.608845 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 02:17:55.612137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612119 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 02:17:55.612288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612278 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 02:17:55.612322 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612312 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 07 02:17:55.612361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612324 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 02:17:55.612361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612341 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 02:17:55.612361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612353 2564 kubelet.go:398] "Adding apiserver pod source" Jul 07 02:17:55.612465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612363 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 02:17:55.612800 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612767 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 02:17:55.612846 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.612808 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 02:17:55.615194 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.615168 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 02:17:55.617058 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.617038 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 02:17:55.617120 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.617075 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 02:17:55.618145 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618129 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618154 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618165 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618171 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618180 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618185 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618193 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618199 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618206 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618211 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618221 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 02:17:55.618246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.618226 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 02:17:55.619053 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.619040 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 02:17:55.619053 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.619053 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 02:17:55.621400 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.621374 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm6fj" Jul 07 02:17:55.623044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.623027 2564 server.go:1268] "Started kubelet" Jul 07 02:17:55.623213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.623189 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 02:17:55.623290 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.623214 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 02:17:55.623290 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.623223 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 02:17:55.623290 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.623269 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 02:17:55.623431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.623264 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 02:17:55.623431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.623329 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 02:17:55.624101 ip-10-0-137-109 systemd[1]: Started Kubernetes Kubelet. Jul 07 02:17:55.625604 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.625582 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 02:17:55.625686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.625641 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 07 02:17:55.629369 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.629337 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 02:17:55.629859 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.629834 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rm6fj" Jul 07 02:17:55.631578 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.631552 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 02:17:55.631578 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.631566 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 02:17:55.631758 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.631742 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:55.632152 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.632134 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 02:17:55.632308 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.632137 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 02:17:55.632308 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.632309 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 02:17:55.632470 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.632432 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 07 02:17:55.632470 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.632440 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 07 02:17:55.634094 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.629611 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-109.ec2.internal.18bfe05ee4408b57 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-109.ec2.internal,UID:ip-10-0-137-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-109.ec2.internal,},FirstTimestamp:2026-07-07 02:17:55.622988631 +0000 UTC m=+0.380537648,LastTimestamp:2026-07-07 02:17:55.622988631 +0000 UTC m=+0.380537648,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-109.ec2.internal,}" Jul 07 02:17:55.634094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.633951 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 02:17:55.634094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.633976 2564 factory.go:55] Registering systemd factory Jul 07 02:17:55.634094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.633985 2564 factory.go:223] Registration of the systemd container factory successfully Jul 07 02:17:55.634477 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.634457 2564 factory.go:153] Registering CRI-O factory Jul 07 02:17:55.634548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.634482 2564 factory.go:223] Registration of the crio container factory successfully Jul 07 02:17:55.634548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.634510 2564 factory.go:103] Registering Raw factory Jul 07 02:17:55.634548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.634527 2564 manager.go:1196] Started watching for new ooms in manager Jul 07 02:17:55.635511 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.635409 2564 manager.go:349] Starting recovery of all containers Jul 07 02:17:55.640015 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.639968 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 02:17:55.640015 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.639996 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 02:17:55.645495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.645469 2564 manager.go:354] Recovery completed Jul 07 02:17:55.647004 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.646973 2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 02:17:55.650567 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.650541 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.652847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.652823 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.652927 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.652866 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.652927 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.652880 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.653546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.653531 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 02:17:55.653546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.653541 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.653628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.653545 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 02:17:55.653628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.653608 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 07 02:17:55.654375 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654359 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.654440 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654392 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.654440 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654402 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.654963 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654950 2564 policy_none.go:49] "None policy: Start" Jul 07 02:17:55.655006 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654968 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 02:17:55.655006 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.654979 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 02:17:55.656172 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.656159 2564 policy_none.go:47] "Start" Jul 07 02:17:55.689602 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.689579 2564 manager.go:337] "Starting Device Plugin manager" Jul 07 02:17:55.689766 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.689617 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 02:17:55.689766 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.689628 2564 server.go:87] "Starting device plugin registration server" Jul 07 02:17:55.689989 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.689976 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 02:17:55.690056 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.689995 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 02:17:55.690111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.690092 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 02:17:55.690197 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.690185 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 02:17:55.690247 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.690198 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 02:17:55.690827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.690802 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 02:17:55.690947 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.690860 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:55.783461 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.783412 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 02:17:55.784908 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.784887 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 02:17:55.784993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.784940 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 02:17:55.784993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.784970 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 02:17:55.785086 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.785017 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 02:17:55.787754 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.787691 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:55.791109 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.791091 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.792230 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.792214 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.792335 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.792252 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.792335 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.792265 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.792335 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.792295 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.801615 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.801580 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.801734 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.801626 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-109.ec2.internal\": node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:55.821839 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.821665 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:55.886090 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.885995 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal"] Jul 07 02:17:55.886263 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.886148 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.887289 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.887261 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.887356 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.887298 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.887356 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.887309 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.888643 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.888627 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.888757 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.888742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.888815 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.888774 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.889488 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889468 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.889571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889499 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.889571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889511 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.889571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889527 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.889571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889551 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.889571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.889565 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.891522 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.891504 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.891601 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.891536 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 02:17:55.892457 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.892438 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 02:17:55.892557 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.892472 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 02:17:55.892557 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:55.892483 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 02:17:55.922473 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.922442 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:55.922686 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.922670 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-109.ec2.internal\" not found" node="ip-10-0-137-109.ec2.internal" Jul 07 02:17:55.927756 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:55.927736 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-109.ec2.internal\" not found" node="ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.023395 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.023351 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.033709 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.033671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.033851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.033713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8c8e4db5eb6f333e24623a8dc7ce358-config\") pod \"kube-apiserver-proxy-ip-10-0-137-109.ec2.internal\" (UID: \"a8c8e4db5eb6f333e24623a8dc7ce358\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.033851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.033742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.124244 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.124163 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.134608 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.134704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.134704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8c8e4db5eb6f333e24623a8dc7ce358-config\") pod \"kube-apiserver-proxy-ip-10-0-137-109.ec2.internal\" (UID: \"a8c8e4db5eb6f333e24623a8dc7ce358\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.134704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134691 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8c8e4db5eb6f333e24623a8dc7ce358-config\") pod \"kube-apiserver-proxy-ip-10-0-137-109.ec2.internal\" (UID: \"a8c8e4db5eb6f333e24623a8dc7ce358\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.134846 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.134846 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.134739 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c8600e9d6ee8c59288b7af349b705f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal\" (UID: \"8c8600e9d6ee8c59288b7af349b705f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.224774 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.224718 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.225868 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.225849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.231955 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.231934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.325844 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.325800 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.426468 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.426367 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.526902 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.526861 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.565346 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.565306 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 02:17:56.565937 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.565495 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 02:17:56.627805 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.627752 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.638352 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.638316 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:12:55 +0000 UTC" deadline="2028-02-15 15:26:57.823233631 +0000 UTC" Jul 07 02:17:56.638352 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.638346 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14125h9m1.18488993s" Jul 07 02:17:56.655609 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.655573 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:56.728772 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.728686 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.760938 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:56.760896 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8600e9d6ee8c59288b7af349b705f2.slice/crio-6df249d8cd93d047521c84f85fcc2d01ea4a5398b9714e0652688bc0806f174a WatchSource:0}: Error finding container 6df249d8cd93d047521c84f85fcc2d01ea4a5398b9714e0652688bc0806f174a: Status 404 returned error can't find the container with id 6df249d8cd93d047521c84f85fcc2d01ea4a5398b9714e0652688bc0806f174a Jul 07 02:17:56.767276 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.767191 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:17:56.788918 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.788868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" event={"ID":"a8c8e4db5eb6f333e24623a8dc7ce358","Type":"ContainerStarted","Data":"0f5c7e276e25efd396464552fa5af7b76b6119b2c40e57b1985c3852ec568469"} Jul 07 02:17:56.789880 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.789855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" event={"ID":"8c8600e9d6ee8c59288b7af349b705f2","Type":"ContainerStarted","Data":"6df249d8cd93d047521c84f85fcc2d01ea4a5398b9714e0652688bc0806f174a"} Jul 07 02:17:56.829018 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:56.828981 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-109.ec2.internal\" not found" Jul 07 02:17:56.837996 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.837970 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:56.932223 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.932195 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.943198 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.943177 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 02:17:56.944204 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.944187 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" Jul 07 02:17:56.950431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.950403 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 02:17:56.985795 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:56.985703 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:57.613434 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.613396 2564 apiserver.go:52] "Watching apiserver" Jul 07 02:17:57.622981 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.622949 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 02:17:57.625519 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.625479 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lc69h","kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27","openshift-cluster-node-tuning-operator/tuned-96tgp","openshift-image-registry/node-ca-j96jg","openshift-multus/multus-z5h7d","openshift-network-operator/iptables-alerter-drcx9","kube-system/global-pull-secret-syncer-zj6zc","openshift-dns/node-resolver-lq4gw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-5zxmf","openshift-multus/network-metrics-daemon-t4vsh","openshift-network-diagnostics/network-check-target-f85jb","openshift-ovn-kubernetes/ovnkube-node-qz6ck"] Jul 07 02:17:57.627607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.627585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.629025 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.628746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:57.629025 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.628844 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:17:57.629896 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.629874 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.630373 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.630350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.630545 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.630524 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 02:17:57.630763 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.630720 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.631075 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.631057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pscfz\"" Jul 07 02:17:57.631976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.631943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.632480 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.632457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 02:17:57.633514 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.633493 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2c4p\"" Jul 07 02:17:57.633626 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.633610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.634059 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.634040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.634692 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.634672 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.636221 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.635034 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 02:17:57.636221 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.636167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.637898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.636457 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 02:17:57.637898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.637106 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.637898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.637345 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.637898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.637426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.638307 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.638290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mqm9k\"" Jul 07 02:17:57.638442 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.638425 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.638552 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.638530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.638636 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.638379 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j5zdj\"" Jul 07 02:17:57.639226 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.639171 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.639226 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.639190 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.639313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.639228 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zldg6\"" Jul 07 02:17:57.639313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.639174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.639313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.639287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 02:17:57.640460 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.640446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.640531 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.640514 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 02:17:57.641237 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.641165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w25kw\"" Jul 07 02:17:57.641545 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.641529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.642007 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.641988 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6ptz\"" Jul 07 02:17:57.642142 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-tmp\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642191 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642191 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-conf\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642255 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-host\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642255 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-socket-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642255 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/783ad7e7-49ea-45b5-a68e-d4c5fea02072-host-slash\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-registration-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-var-lib-kubelet\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mvg\" (UniqueName: \"kubernetes.io/projected/f500c04b-a04c-4a18-87e0-9708bbbace54-kube-api-access-d6mvg\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b434ae8a-907e-4260-aa20-16ba66f6e467-host\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b434ae8a-907e-4260-aa20-16ba66f6e467-serviceca\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxz6\" (UniqueName: \"kubernetes.io/projected/b16307ca-d753-4ee4-be19-f790c4abc957-kube-api-access-mqxz6\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-run\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysconfig\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-modprobe-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-systemd\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-lib-modules\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/783ad7e7-49ea-45b5-a68e-d4c5fea02072-iptables-alerter-script\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-kubernetes\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-sys\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-tuned\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbgh\" (UniqueName: \"kubernetes.io/projected/b434ae8a-907e-4260-aa20-16ba66f6e467-kube-api-access-6qbgh\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4bc\" (UniqueName: \"kubernetes.io/projected/783ad7e7-49ea-45b5-a68e-d4c5fea02072-kube-api-access-pj4bc\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642599 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-sys-fs\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.642744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-device-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.643326 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 02:17:57.643326 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.642774 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.643414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.643369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 02:17:57.643506 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.643485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 02:17:57.643704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.643685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.643704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.643700 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 02:17:57.643868 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.643760 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:17:57.643946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.643879 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 02:17:57.644146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.644124 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 02:17:57.644211 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.644153 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 02:17:57.644360 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.644321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-284ss\"" Jul 07 02:17:57.645816 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.645193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.645816 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.645267 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 02:17:57.646819 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.646766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.646937 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.646862 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:17:57.647656 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.647626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 02:17:57.647931 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.647912 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vzl5g\"" Jul 07 02:17:57.648055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.647990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 02:17:57.668864 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.668836 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl7sq" Jul 07 02:17:57.676194 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.676159 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl7sq" Jul 07 02:17:57.710433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.710408 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 02:17:57.733590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.733554 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 02:17:57.742906 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.742874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.742913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/783ad7e7-49ea-45b5-a68e-d4c5fea02072-host-slash\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.742943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-node-log\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.742966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.742992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27712a9b-7072-4450-85c5-abb283829105-agent-certs\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cni-binary-copy\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-conf-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-registration-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-systemd-units\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.743096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743091 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-slash\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-etc-kubernetes\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-systemd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743174 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/783ad7e7-49ea-45b5-a68e-d4c5fea02072-host-slash\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-os-release\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-config\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-system-cni-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysconfig\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743309 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-registration-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysconfig\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-modprobe-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-systemd\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-lib-modules\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.743495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-kubernetes\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743490 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-modprobe-d\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-sys\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743531 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-tuned\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4bc\" (UniqueName: \"kubernetes.io/projected/783ad7e7-49ea-45b5-a68e-d4c5fea02072-kube-api-access-pj4bc\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-lib-modules\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-netns\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-log-socket\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-sys\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4t6\" (UniqueName: \"kubernetes.io/projected/e3bc19ab-4f53-46d0-958e-52ff15ae512e-kube-api-access-mq4t6\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cnibin\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-kubelet\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-socket-dir-parent\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-hostroot\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-daemon-config\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-multus-certs\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743583 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-kubernetes\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743940 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-host\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.743994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-kubelet-config\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-bin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-socket-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-kubelet\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-ovn\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744089 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744218 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-socket-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744298 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-systemd\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.744837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744341 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-host\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46sxq\" (UniqueName: \"kubernetes.io/projected/b7b9e554-ea7a-4647-b208-0cbb59badeb6-kube-api-access-46sxq\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-dbus\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cnibin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/985a2683-98de-489c-9990-6960d54df436-ovn-node-metrics-cert\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-var-lib-kubelet\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mvg\" (UniqueName: \"kubernetes.io/projected/f500c04b-a04c-4a18-87e0-9708bbbace54-kube-api-access-d6mvg\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b434ae8a-907e-4260-aa20-16ba66f6e467-host\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b434ae8a-907e-4260-aa20-16ba66f6e467-serviceca\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a946adcf-84da-4601-a697-821c2c425a08-tmp-dir\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxrl\" (UniqueName: \"kubernetes.io/projected/a946adcf-84da-4601-a697-821c2c425a08-kube-api-access-zzxrl\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744579 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-var-lib-kubelet\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxz6\" (UniqueName: \"kubernetes.io/projected/b16307ca-d753-4ee4-be19-f790c4abc957-kube-api-access-mqxz6\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744737 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b434ae8a-907e-4260-aa20-16ba66f6e467-host\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-run\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-run\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.745590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-var-lib-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-netd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-k8s-cni-cncf-io\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-multus\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-bin\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-env-overrides\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.744994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/783ad7e7-49ea-45b5-a68e-d4c5fea02072-iptables-alerter-script\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-system-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-netns\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5cw\" (UniqueName: \"kubernetes.io/projected/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-kube-api-access-lk5cw\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbgh\" (UniqueName: \"kubernetes.io/projected/b434ae8a-907e-4260-aa20-16ba66f6e467-kube-api-access-6qbgh\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745215 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b434ae8a-907e-4260-aa20-16ba66f6e467-serviceca\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a946adcf-84da-4601-a697-821c2c425a08-hosts-file\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-os-release\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-sys-fs\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-script-lib\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.746532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-device-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-sys-fs\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745465 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b16307ca-d753-4ee4-be19-f790c4abc957-device-dir\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-tmp\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-etc-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27712a9b-7072-4450-85c5-abb283829105-konnectivity-ca\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-conf\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745574 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/783ad7e7-49ea-45b5-a68e-d4c5fea02072-iptables-alerter-script\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9nb4\" (UniqueName: \"kubernetes.io/projected/985a2683-98de-489c-9990-6960d54df436-kube-api-access-c9nb4\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.747392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.745702 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-sysctl-conf\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.747856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.747732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-etc-tuned\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.748112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.748090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f500c04b-a04c-4a18-87e0-9708bbbace54-tmp\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.754218 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.754184 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:57.754218 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.754220 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:57.754426 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.754235 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:57.754426 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.754380 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:58.254350743 +0000 UTC m=+3.011899760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:57.756711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.756686 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4bc\" (UniqueName: \"kubernetes.io/projected/783ad7e7-49ea-45b5-a68e-d4c5fea02072-kube-api-access-pj4bc\") pod \"iptables-alerter-drcx9\" (UID: \"783ad7e7-49ea-45b5-a68e-d4c5fea02072\") " pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.757898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.757872 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbgh\" (UniqueName: \"kubernetes.io/projected/b434ae8a-907e-4260-aa20-16ba66f6e467-kube-api-access-6qbgh\") pod \"node-ca-j96jg\" (UID: \"b434ae8a-907e-4260-aa20-16ba66f6e467\") " pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.758300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.758280 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mvg\" (UniqueName: \"kubernetes.io/projected/f500c04b-a04c-4a18-87e0-9708bbbace54-kube-api-access-d6mvg\") pod \"tuned-96tgp\" (UID: \"f500c04b-a04c-4a18-87e0-9708bbbace54\") " pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.761600 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.761570 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxz6\" (UniqueName: \"kubernetes.io/projected/b16307ca-d753-4ee4-be19-f790c4abc957-kube-api-access-mqxz6\") pod \"aws-ebs-csi-driver-node-d7l27\" (UID: \"b16307ca-d753-4ee4-be19-f790c4abc957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.845892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.845931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-socket-dir-parent\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.845956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-hostroot\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.845962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.845988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-daemon-config\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-multus-certs\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846030 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-socket-dir-parent\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-kubelet-config\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846014 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-hostroot\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-bin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846082 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-multus-certs\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-kubelet\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846131 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-bin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-ovn\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46sxq\" (UniqueName: \"kubernetes.io/projected/b7b9e554-ea7a-4647-b208-0cbb59badeb6-kube-api-access-46sxq\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-kubelet\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-dbus\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846264 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-ovn\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cnibin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/985a2683-98de-489c-9990-6960d54df436-ovn-node-metrics-cert\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a946adcf-84da-4601-a697-821c2c425a08-tmp-dir\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxrl\" (UniqueName: \"kubernetes.io/projected/a946adcf-84da-4601-a697-821c2c425a08-kube-api-access-zzxrl\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-var-lib-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846403 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-netd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846428 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-dbus\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846442 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cnibin\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-k8s-cni-cncf-io\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846481 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-var-lib-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-multus\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-bin\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-env-overrides\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.847829 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-system-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-netd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-netns\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5cw\" (UniqueName: \"kubernetes.io/projected/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-kube-api-access-lk5cw\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846624 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-k8s-cni-cncf-io\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a946adcf-84da-4601-a697-821c2c425a08-hosts-file\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-os-release\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846664 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-daemon-config\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-cni-bin\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846691 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a946adcf-84da-4601-a697-821c2c425a08-tmp-dir\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-run-netns\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-script-lib\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846726 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40706701-1bf7-40be-b011-fb80c37fe4c1-kubelet-config\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846745 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-etc-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-os-release\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27712a9b-7072-4450-85c5-abb283829105-konnectivity-ca\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.848577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846838 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9nb4\" (UniqueName: \"kubernetes.io/projected/985a2683-98de-489c-9990-6960d54df436-kube-api-access-c9nb4\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-node-log\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a946adcf-84da-4601-a697-821c2c425a08-hosts-file\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-cni-multus\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27712a9b-7072-4450-85c5-abb283829105-agent-certs\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cni-binary-copy\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-conf-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-systemd-units\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-slash\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-etc-kubernetes\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-systemd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847110 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-env-overrides\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-systemd-units\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-os-release\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.849372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-config\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.847219 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-system-cni-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.847277 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:58.347261072 +0000 UTC m=+3.104810077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847250 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-node-log\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-netns\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847362 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27712a9b-7072-4450-85c5-abb283829105-konnectivity-ca\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847329 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-slash\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847362 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.846759 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-system-cni-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-host-run-netns\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-run-systemd\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-log-socket\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-log-socket\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847436 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/985a2683-98de-489c-9990-6960d54df436-etc-openvswitch\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847476 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-multus-conf-dir\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4t6\" (UniqueName: \"kubernetes.io/projected/e3bc19ab-4f53-46d0-958e-52ff15ae512e-kube-api-access-mq4t6\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-system-cni-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cnibin\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-os-release\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847829 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-etc-kubernetes\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847872 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cnibin\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.847903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-kubelet\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.847995 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.848025 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-host-var-lib-kubelet\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.848032 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3bc19ab-4f53-46d0-958e-52ff15ae512e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:57.848075 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:58.348059921 +0000 UTC m=+3.105608942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.848026 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-cni-binary-copy\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.848205 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-script-lib\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.850962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.848884 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/985a2683-98de-489c-9990-6960d54df436-ovnkube-config\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.851679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.849321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/985a2683-98de-489c-9990-6960d54df436-ovn-node-metrics-cert\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.852883 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.852843 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27712a9b-7072-4450-85c5-abb283829105-agent-certs\") pod \"konnectivity-agent-lc69h\" (UID: \"27712a9b-7072-4450-85c5-abb283829105\") " pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.855866 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.855756 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5cw\" (UniqueName: \"kubernetes.io/projected/b84a7f82-3bde-45ad-a2ff-7cfe221b1037-kube-api-access-lk5cw\") pod \"multus-z5h7d\" (UID: \"b84a7f82-3bde-45ad-a2ff-7cfe221b1037\") " pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.856439 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.856189 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxrl\" (UniqueName: \"kubernetes.io/projected/a946adcf-84da-4601-a697-821c2c425a08-kube-api-access-zzxrl\") pod \"node-resolver-lq4gw\" (UID: \"a946adcf-84da-4601-a697-821c2c425a08\") " pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:57.856549 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.856443 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9nb4\" (UniqueName: \"kubernetes.io/projected/985a2683-98de-489c-9990-6960d54df436-kube-api-access-c9nb4\") pod \"ovnkube-node-qz6ck\" (UID: \"985a2683-98de-489c-9990-6960d54df436\") " pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:57.856861 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.856838 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46sxq\" (UniqueName: \"kubernetes.io/projected/b7b9e554-ea7a-4647-b208-0cbb59badeb6-kube-api-access-46sxq\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:57.857497 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.857476 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4t6\" (UniqueName: \"kubernetes.io/projected/e3bc19ab-4f53-46d0-958e-52ff15ae512e-kube-api-access-mq4t6\") pod \"multus-additional-cni-plugins-5zxmf\" (UID: \"e3bc19ab-4f53-46d0-958e-52ff15ae512e\") " pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:57.945839 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.945740 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-drcx9" Jul 07 02:17:57.954118 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.954087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" Jul 07 02:17:57.964968 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.964931 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-96tgp" Jul 07 02:17:57.971028 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.971001 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j96jg" Jul 07 02:17:57.981066 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.981034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5h7d" Jul 07 02:17:57.990018 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.989985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:17:57.997207 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:57.997175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq4gw" Jul 07 02:17:58.006217 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.006191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:17:58.011146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.011117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" Jul 07 02:17:58.351575 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.351465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:58.351575 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.351511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.351583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351696 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351707 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351728 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351742 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351700 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:58.351827 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351799 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:59.351765357 +0000 UTC m=+4.109314360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:58.352086 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351847 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:59.35183098 +0000 UTC m=+4.109379988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:58.352086 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.351863 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:59.351853656 +0000 UTC m=+4.109402666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:58.672221 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.670059 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16307ca_d753_4ee4_be19_f790c4abc957.slice/crio-25afe5cda8488adccf39f875c2317bb456222ac7670e928190533042f4a19ec6 WatchSource:0}: Error finding container 25afe5cda8488adccf39f875c2317bb456222ac7670e928190533042f4a19ec6: Status 404 returned error can't find the container with id 25afe5cda8488adccf39f875c2317bb456222ac7670e928190533042f4a19ec6 Jul 07 02:17:58.673001 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.672934 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84a7f82_3bde_45ad_a2ff_7cfe221b1037.slice/crio-59f75ebb354967d9a1677c297f6e2bd7e1e48a912090446d938175f2d7f7f75f WatchSource:0}: Error finding container 59f75ebb354967d9a1677c297f6e2bd7e1e48a912090446d938175f2d7f7f75f: Status 404 returned error can't find the container with id 59f75ebb354967d9a1677c297f6e2bd7e1e48a912090446d938175f2d7f7f75f Jul 07 02:17:58.678418 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.678374 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:57 +0000 UTC" deadline="2027-12-13 01:27:51.831975512 +0000 UTC" Jul 07 02:17:58.678418 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.678417 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12575h9m53.153561712s" Jul 07 02:17:58.679067 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.679024 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf500c04b_a04c_4a18_87e0_9708bbbace54.slice/crio-f9b3fd74520f2d0c3f7d3dd4dd2a173756a493e4d46cf94ecc33174e720eb6b8 WatchSource:0}: Error finding container f9b3fd74520f2d0c3f7d3dd4dd2a173756a493e4d46cf94ecc33174e720eb6b8: Status 404 returned error can't find the container with id f9b3fd74520f2d0c3f7d3dd4dd2a173756a493e4d46cf94ecc33174e720eb6b8 Jul 07 02:17:58.679939 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.679918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bc19ab_4f53_46d0_958e_52ff15ae512e.slice/crio-ab96d14ee03a909597d43c408cbe0184b74eb0f2727104ba5545b07f81e89d08 WatchSource:0}: Error finding container ab96d14ee03a909597d43c408cbe0184b74eb0f2727104ba5545b07f81e89d08: Status 404 returned error can't find the container with id ab96d14ee03a909597d43c408cbe0184b74eb0f2727104ba5545b07f81e89d08 Jul 07 02:17:58.681810 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.681419 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985a2683_98de_489c_9990_6960d54df436.slice/crio-c69dc9e71a2eb68c38aeeaac788e8b4cc78332b89a859dad16a2308c75bc9387 WatchSource:0}: Error finding container c69dc9e71a2eb68c38aeeaac788e8b4cc78332b89a859dad16a2308c75bc9387: Status 404 returned error can't find the container with id c69dc9e71a2eb68c38aeeaac788e8b4cc78332b89a859dad16a2308c75bc9387 Jul 07 02:17:58.682493 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.682403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783ad7e7_49ea_45b5_a68e_d4c5fea02072.slice/crio-0c367542514f8f38716e04d8e7412b49e8994717455e4c8a8145fa3dffc154ed WatchSource:0}: Error finding container 0c367542514f8f38716e04d8e7412b49e8994717455e4c8a8145fa3dffc154ed: Status 404 returned error can't find the container with id 0c367542514f8f38716e04d8e7412b49e8994717455e4c8a8145fa3dffc154ed Jul 07 02:17:58.683972 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.683916 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb434ae8a_907e_4260_aa20_16ba66f6e467.slice/crio-63cc43618c5670109f206f344b3ac84de1c8ea5455fe54285a5b8db79522961c WatchSource:0}: Error finding container 63cc43618c5670109f206f344b3ac84de1c8ea5455fe54285a5b8db79522961c: Status 404 returned error can't find the container with id 63cc43618c5670109f206f344b3ac84de1c8ea5455fe54285a5b8db79522961c Jul 07 02:17:58.684403 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:17:58.684323 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27712a9b_7072_4450_85c5_abb283829105.slice/crio-83822ca48aa7618cd2bb0859eb6590f5e00398ee85259d8b16cedb5d497449cc WatchSource:0}: Error finding container 83822ca48aa7618cd2bb0859eb6590f5e00398ee85259d8b16cedb5d497449cc: Status 404 returned error can't find the container with id 83822ca48aa7618cd2bb0859eb6590f5e00398ee85259d8b16cedb5d497449cc Jul 07 02:17:58.786053 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.786016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:58.786192 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:58.786133 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:17:58.796362 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.796325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq4gw" event={"ID":"a946adcf-84da-4601-a697-821c2c425a08","Type":"ContainerStarted","Data":"24c82fd334ac14d59aaef52152902d6e8c9700a7c1d8c4d474af1b450f683af3"} Jul 07 02:17:58.800377 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.800340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5h7d" event={"ID":"b84a7f82-3bde-45ad-a2ff-7cfe221b1037","Type":"ContainerStarted","Data":"59f75ebb354967d9a1677c297f6e2bd7e1e48a912090446d938175f2d7f7f75f"} Jul 07 02:17:58.804856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.804817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j96jg" event={"ID":"b434ae8a-907e-4260-aa20-16ba66f6e467","Type":"ContainerStarted","Data":"63cc43618c5670109f206f344b3ac84de1c8ea5455fe54285a5b8db79522961c"} Jul 07 02:17:58.807014 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.806983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96tgp" event={"ID":"f500c04b-a04c-4a18-87e0-9708bbbace54","Type":"ContainerStarted","Data":"f9b3fd74520f2d0c3f7d3dd4dd2a173756a493e4d46cf94ecc33174e720eb6b8"} Jul 07 02:17:58.809823 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.809771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" event={"ID":"b16307ca-d753-4ee4-be19-f790c4abc957","Type":"ContainerStarted","Data":"25afe5cda8488adccf39f875c2317bb456222ac7670e928190533042f4a19ec6"} Jul 07 02:17:58.811761 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.811730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" event={"ID":"a8c8e4db5eb6f333e24623a8dc7ce358","Type":"ContainerStarted","Data":"7e59b1612d2cee3772cd1882ea2810f62812e227aa015433a6fd5c341347613c"} Jul 07 02:17:58.812981 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.812943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc69h" event={"ID":"27712a9b-7072-4450-85c5-abb283829105","Type":"ContainerStarted","Data":"83822ca48aa7618cd2bb0859eb6590f5e00398ee85259d8b16cedb5d497449cc"} Jul 07 02:17:58.815023 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.814997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-drcx9" event={"ID":"783ad7e7-49ea-45b5-a68e-d4c5fea02072","Type":"ContainerStarted","Data":"0c367542514f8f38716e04d8e7412b49e8994717455e4c8a8145fa3dffc154ed"} Jul 07 02:17:58.817414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.817390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"c69dc9e71a2eb68c38aeeaac788e8b4cc78332b89a859dad16a2308c75bc9387"} Jul 07 02:17:58.820212 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.820185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerStarted","Data":"ab96d14ee03a909597d43c408cbe0184b74eb0f2727104ba5545b07f81e89d08"} Jul 07 02:17:58.826722 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:58.826665 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-109.ec2.internal" podStartSLOduration=2.826648625 podStartE2EDuration="2.826648625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:17:58.826079628 +0000 UTC m=+3.583628653" watchObservedRunningTime="2026-07-07 02:17:58.826648625 +0000 UTC m=+3.584197650" Jul 07 02:17:59.366720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.366612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:59.366720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.366699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:59.366979 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.366730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:17:59.366979 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.366895 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:59.366979 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.366957 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:01.366938197 +0000 UTC m=+6.124487205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:17:59.367442 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367419 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:59.367553 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367496 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:01.367462153 +0000 UTC m=+6.125011160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:17:59.367620 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367576 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:17:59.367620 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367590 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:17:59.367620 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367604 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:59.367763 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.367641 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:01.367629016 +0000 UTC m=+6.125178025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:17:59.679066 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.678930 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:57 +0000 UTC" deadline="2028-02-13 04:54:46.651648132 +0000 UTC" Jul 07 02:17:59.679066 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.678999 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14066h36m46.97265449s" Jul 07 02:17:59.788094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.788057 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:17:59.788287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.788200 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:17:59.788660 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.788637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:17:59.788822 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:17:59.788796 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:17:59.834775 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.834155 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c8600e9d6ee8c59288b7af349b705f2" containerID="cc8cc2ee84f8ad65997d71d6be6d77429f8d6216d91761946f6e89c531829684" exitCode=0 Jul 07 02:17:59.834775 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:17:59.834721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" event={"ID":"8c8600e9d6ee8c59288b7af349b705f2","Type":"ContainerDied","Data":"cc8cc2ee84f8ad65997d71d6be6d77429f8d6216d91761946f6e89c531829684"} Jul 07 02:18:00.785989 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:00.785952 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:00.786507 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:00.786097 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:00.873872 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:00.873832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" event={"ID":"8c8600e9d6ee8c59288b7af349b705f2","Type":"ContainerStarted","Data":"e58ee5cebbe71099dc5573e7b58774476ba58a844b19b6d0d3469e20eb554b5d"} Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:01.383841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:01.383916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:01.383957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384111 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384179 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:05.384160698 +0000 UTC m=+10.141709707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384580 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384631 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:05.384614919 +0000 UTC m=+10.142163925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384706 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384721 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384733 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:01.384824 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.384768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:05.384756258 +0000 UTC m=+10.142305266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:01.786994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:01.786957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:01.787541 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.787120 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:01.787541 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:01.787520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:01.787644 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:01.787621 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:02.785632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:02.785594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:02.785833 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:02.785729 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:03.787976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:03.787928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:03.788440 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:03.788076 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:03.788440 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:03.788084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:03.788440 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:03.788191 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:04.785287 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:04.785246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:04.785505 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:04.785419 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:05.418170 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:05.418130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:05.418180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:05.418198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418310 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418359 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.418345892 +0000 UTC m=+18.175894896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418401 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418421 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.418413892 +0000 UTC m=+18.175962895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418474 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418484 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418493 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:05.418590 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.418514 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.418507376 +0000 UTC m=+18.176056379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:05.786969 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:05.786932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:05.787130 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.787049 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:05.787431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:05.787414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:05.787515 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:05.787500 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:06.786304 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:06.785810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:06.786304 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:06.785959 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:07.786366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:07.786329 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:07.786889 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:07.786466 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:07.786961 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:07.786939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:07.787074 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:07.787051 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:08.786034 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:08.785999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:08.786241 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:08.786131 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:09.785686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:09.785647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:09.786192 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:09.785653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:09.786192 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:09.785821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:09.786192 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:09.785878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:10.785817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:10.785767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:10.786264 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:10.785899 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:11.785846 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:11.785808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:11.786295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:11.785815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:11.786295 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:11.785961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:11.786295 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:11.786026 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:12.785397 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:12.785359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:12.785587 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:12.785487 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:13.483772 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:13.483731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:13.483809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:13.483835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.483918 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.483949 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.483991 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.484003 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:29.483982137 +0000 UTC m=+34.241531140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.484019 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.484026 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:29.484016593 +0000 UTC m=+34.241565602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.484035 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:13.484287 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.484071 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:29.484058288 +0000 UTC m=+34.241607290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:13.785539 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:13.785491 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:13.785715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:13.785628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:13.785715 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.785634 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:13.785715 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:13.785707 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:14.785596 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:14.785556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:14.786078 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:14.785692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:15.786283 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:15.786245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:15.786750 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:15.786357 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:15.786750 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:15.786410 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:15.786936 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:15.786908 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:16.785858 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:16.785812 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:16.786041 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:16.785961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:17.785618 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:17.785582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:17.785618 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:17.785617 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:17.786075 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:17.785708 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:17.786075 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:17.785840 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:18.785712 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.785376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:18.786398 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:18.785812 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:18.916517 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.916230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lc69h" event={"ID":"27712a9b-7072-4450-85c5-abb283829105","Type":"ContainerStarted","Data":"6a33429dbee093146ec305657dfbe793e9588929297133251f2f10103f593bfd"} Jul 07 02:18:18.919774 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.919746 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:18:18.920203 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920177 2564 generic.go:356] "Generic (PLEG): container finished" podID="985a2683-98de-489c-9990-6960d54df436" containerID="b3cd4a760cc71ecf9d64f621dd12883c3ea5d676fee9d367e2b0d618cbf3cde5" exitCode=1 Jul 07 02:18:18.920300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"c6628a6142c345ff178cc94bcbeac5fddee993d70470d51c3301e8040ae2aa40"} Jul 07 02:18:18.920361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"e1ccfc3f6285ba654a123189f3c8e3b949b3070693dc6cff07b8d766dcf55037"} Jul 07 02:18:18.920450 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"fedce4911449f7d4d251c4b2a5dea2dd1d94ee11c3acac90e6b93d39d1d2f023"} Jul 07 02:18:18.920507 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"8fc5b53617084b76e7e7efebb8b5533711ebb305117f0cfed274d8f54e715304"} Jul 07 02:18:18.920551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerDied","Data":"b3cd4a760cc71ecf9d64f621dd12883c3ea5d676fee9d367e2b0d618cbf3cde5"} Jul 07 02:18:18.920605 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.920590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"3aaa380072e58f1bb840d9422e661ffc26abb52ad08536ba3a50ad0bb81cc7ab"} Jul 07 02:18:18.922143 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.922118 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="200bd3c367af8c398f2161569aa97658e6bfaae8133b95641287d5c9dc71fa36" exitCode=0 Jul 07 02:18:18.922267 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.922203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"200bd3c367af8c398f2161569aa97658e6bfaae8133b95641287d5c9dc71fa36"} Jul 07 02:18:18.923859 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.923832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq4gw" event={"ID":"a946adcf-84da-4601-a697-821c2c425a08","Type":"ContainerStarted","Data":"8bd72639dede3c3167a0810678b6dc23395c2a658f2ebbd4113a38becc7b5839"} Jul 07 02:18:18.925576 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.925535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5h7d" event={"ID":"b84a7f82-3bde-45ad-a2ff-7cfe221b1037","Type":"ContainerStarted","Data":"f63f927fa23de909cdf5723779d0e99de112647551037ec4d5af4ec2bb7e6a7b"} Jul 07 02:18:18.927265 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.927238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j96jg" event={"ID":"b434ae8a-907e-4260-aa20-16ba66f6e467","Type":"ContainerStarted","Data":"bbf5d7d186a710144d73ee3cab9fd120c6487b113257ebc4dc20a57a4eb101e1"} Jul 07 02:18:18.928972 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.928944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-96tgp" event={"ID":"f500c04b-a04c-4a18-87e0-9708bbbace54","Type":"ContainerStarted","Data":"65d4f13632abf822c67af9eaaff31fec6387653b4d8ba7e74a6e18eb1eec6a9b"} Jul 07 02:18:18.930746 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.930717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" event={"ID":"b16307ca-d753-4ee4-be19-f790c4abc957","Type":"ContainerStarted","Data":"e5236d9df0e4f6c5091edcffd1e576e83516815870a2bf8601b4661006d7762c"} Jul 07 02:18:18.935012 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.934965 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-109.ec2.internal" podStartSLOduration=22.9349464 podStartE2EDuration="22.9349464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:00.890848849 +0000 UTC m=+5.648397893" watchObservedRunningTime="2026-07-07 02:18:18.9349464 +0000 UTC m=+23.692495424" Jul 07 02:18:18.935374 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.935342 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lc69h" podStartSLOduration=13.186845388 podStartE2EDuration="23.935334283s" totalImagesPullingTime="10.748488895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:18.933806036 +0000 UTC m=+23.691355075" watchObservedRunningTime="2026-07-07 02:18:18.935334283 +0000 UTC m=+23.692883307" Jul 07 02:18:18.960406 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.960336 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5h7d" podStartSLOduration=4.722257789 podStartE2EDuration="23.960321059s" totalImagesPullingTime="19.23806327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:18.960025234 +0000 UTC m=+23.717574285" watchObservedRunningTime="2026-07-07 02:18:18.960321059 +0000 UTC m=+23.717870190" Jul 07 02:18:18.983885 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:18.983826 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j96jg" podStartSLOduration=13.234654147 podStartE2EDuration="23.983809203s" totalImagesPullingTime="10.749155056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:18.983414027 +0000 UTC m=+23.740963051" watchObservedRunningTime="2026-07-07 02:18:18.983809203 +0000 UTC m=+23.741358219" Jul 07 02:18:19.048150 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.048045 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lq4gw" podStartSLOduration=4.855273359 podStartE2EDuration="24.048027116s" totalImagesPullingTime="19.192753757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:19.015288021 +0000 UTC m=+23.772837046" watchObservedRunningTime="2026-07-07 02:18:19.048027116 +0000 UTC m=+23.805576141" Jul 07 02:18:19.558256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.558027 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 02:18:19.701678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.701565 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:19.558049263Z","UUID":"febfd26a-0378-4871-befa-a3f1c8252d89","Handler":null,"Name":"","Endpoint":""} Jul 07 02:18:19.703723 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.703698 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 02:18:19.703889 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.703745 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 02:18:19.785830 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.785770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:19.786264 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.785800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:19.786264 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:19.785923 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:19.786264 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:19.785986 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:19.935974 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.935834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" event={"ID":"b16307ca-d753-4ee4-be19-f790c4abc957","Type":"ContainerStarted","Data":"40d217d9c7d52fda8618883e41e4777aa0b840db38a415a6f850bb84a17de786"} Jul 07 02:18:19.938237 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.938188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-drcx9" event={"ID":"783ad7e7-49ea-45b5-a68e-d4c5fea02072","Type":"ContainerStarted","Data":"bab42de7ef7adbb754b1a875645b056714afd855b3c38d00d6e958e6a71c45b6"} Jul 07 02:18:19.954972 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.954912 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-96tgp" podStartSLOduration=5.745918914 podStartE2EDuration="24.954893853s" totalImagesPullingTime="19.208974939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:19.079730579 +0000 UTC m=+23.837279604" watchObservedRunningTime="2026-07-07 02:18:19.954893853 +0000 UTC m=+24.712442879" Jul 07 02:18:19.955159 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:19.955123 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-drcx9" podStartSLOduration=5.74990068 podStartE2EDuration="24.955116302s" totalImagesPullingTime="19.205215622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:19.954605466 +0000 UTC m=+24.712154493" watchObservedRunningTime="2026-07-07 02:18:19.955116302 +0000 UTC m=+24.712665327" Jul 07 02:18:20.156466 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.156429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:20.253681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.253578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:20.785656 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.785624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:20.785860 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:20.785743 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:20.944077 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.943970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" event={"ID":"b16307ca-d753-4ee4-be19-f790c4abc957","Type":"ContainerStarted","Data":"0ecea37012664b5ba6970e0fd98df0d75864a6c6942424833b293986eb0bb831"} Jul 07 02:18:20.947720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.947690 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:18:20.948312 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.948136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"96be5209a547f51c182ad70942a21a706373b4460317c4ab81c9f325c60c0c63"} Jul 07 02:18:20.962365 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:20.962260 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7l27" podStartSLOduration=4.70714008 podStartE2EDuration="25.962231766s" totalImagesPullingTime="21.255091686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:20.961596258 +0000 UTC m=+25.719145295" watchObservedRunningTime="2026-07-07 02:18:20.962231766 +0000 UTC m=+25.719780793" Jul 07 02:18:21.786444 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:21.786233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:21.786938 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:21.786558 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:21.787011 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:21.786240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:21.787115 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:21.787088 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:22.785949 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:22.785914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:22.786182 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:22.786044 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:22.832951 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:22.832919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:18:22.833685 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:22.833664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:18:22.953687 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:22.953657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:18:22.954277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:22.954252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lc69h" Jul 07 02:18:23.026681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.026641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:23.785625 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.785332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:23.785838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.785356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:23.785838 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:23.785762 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:23.785949 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:23.785897 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:23.959215 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.959186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:18:23.960175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.959618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"515784df3f6c3c97069f26cfe4cd13c6b4ce0cebc6c2aee69095e5f1565ba91a"} Jul 07 02:18:23.960175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.959991 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:23.960308 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.960244 2564 scope.go:117] "RemoveContainer" containerID="b3cd4a760cc71ecf9d64f621dd12883c3ea5d676fee9d367e2b0d618cbf3cde5" Jul 07 02:18:23.961875 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.961834 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="b1e6f65d664e4e33cd0c908a42c4c836c0d9d356bd116ea1205d49d3a97a44c3" exitCode=0 Jul 07 02:18:23.961875 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.961909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"b1e6f65d664e4e33cd0c908a42c4c836c0d9d356bd116ea1205d49d3a97a44c3"} Jul 07 02:18:23.979621 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:23.979594 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:24.289444 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.289402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:24.390175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.390137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:24.785259 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.785227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:24.785420 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:24.785335 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:24.969296 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.969271 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:18:24.969746 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.969700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" event={"ID":"985a2683-98de-489c-9990-6960d54df436","Type":"ContainerStarted","Data":"aa7e6fab2696e37298975a86552c49dc87dbfae3ae2737938f40bf42a3753eed"} Jul 07 02:18:24.970106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.970087 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:24.970257 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.970238 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:24.972413 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.972382 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="47105175f0540bd712c5b07946f8ca7c617dc5abaa0437976f852184ed7befa1" exitCode=0 Jul 07 02:18:24.972540 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.972461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"47105175f0540bd712c5b07946f8ca7c617dc5abaa0437976f852184ed7befa1"} Jul 07 02:18:24.989633 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.989598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:24.996832 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:24.996763 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" podStartSLOduration=10.7236675 podStartE2EDuration="29.996742828s" totalImagesPullingTime="19.273075328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:24.995156834 +0000 UTC m=+29.752705903" watchObservedRunningTime="2026-07-07 02:18:24.996742828 +0000 UTC m=+29.754291855" Jul 07 02:18:25.181735 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.181704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zj6zc"] Jul 07 02:18:25.181910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.181832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:25.181950 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:25.181925 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:25.185269 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.185237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f85jb"] Jul 07 02:18:25.185422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.185362 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:25.185496 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:25.185459 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:25.185954 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.185920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4vsh"] Jul 07 02:18:25.186049 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.186031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:25.186144 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:25.186124 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:25.401295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.401251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:25.462684 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.462632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:25.519414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.519335 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:25.977708 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.977462 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="609a3dd5111a93498026492ce8e265522772155e8ae539f48560eec381ee3e8a" exitCode=0 Jul 07 02:18:25.978093 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:25.977510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"609a3dd5111a93498026492ce8e265522772155e8ae539f48560eec381ee3e8a"} Jul 07 02:18:26.507474 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:26.507412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:26.610681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:26.610644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:26.785409 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:26.785368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:26.785409 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:26.785403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:26.785625 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:26.785403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:26.785625 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:26.785514 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:26.785704 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:26.785658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:26.785744 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:26.785723 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:28.785334 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:28.785298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:28.786039 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:28.785346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:28.786039 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:28.785346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:28.786039 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:28.785425 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:28.786039 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:28.785530 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:28.786039 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:28.785610 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:29.504175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:29.504134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:29.504387 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:29.504195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:29.504387 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:29.504227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:29.504387 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504349 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:29.504387 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504351 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:29.504387 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504381 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 02:18:29.504598 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504404 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 02:18:29.504598 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504413 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret podName:40706701-1bf7-40be-b011-fb80c37fe4c1 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:01.504399419 +0000 UTC m=+66.261948422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret") pod "global-pull-secret-syncer-zj6zc" (UID: "40706701-1bf7-40be-b011-fb80c37fe4c1") : object "kube-system"/"original-pull-secret" not registered Jul 07 02:18:29.504598 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504419 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mt7nq for pod openshift-network-diagnostics/network-check-target-f85jb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:29.504598 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504432 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:01.504421968 +0000 UTC m=+66.261970973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 02:18:29.504598 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:29.504465 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq podName:739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:01.504448254 +0000 UTC m=+66.261997260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mt7nq" (UniqueName: "kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq") pod "network-check-target-f85jb" (UID: "739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 02:18:30.247961 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.247890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:30.346106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.346063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:30.786267 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.786231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:30.786463 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.786275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:30.786463 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.786371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:30.786463 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:30.786380 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zj6zc" podUID="40706701-1bf7-40be-b011-fb80c37fe4c1" Jul 07 02:18:30.786463 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:30.786437 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f85jb" podUID="739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70" Jul 07 02:18:30.786646 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:30.786566 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:18:30.994327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.994282 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="017f0ccddcb12ceff9413030dd6796fec3d57ba7246b81be28c6d53b0ab74b65" exitCode=0 Jul 07 02:18:30.994509 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:30.994339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"017f0ccddcb12ceff9413030dd6796fec3d57ba7246b81be28c6d53b0ab74b65"} Jul 07 02:18:31.020462 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.020439 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-109.ec2.internal" event="NodeReady" Jul 07 02:18:31.020581 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.020564 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 02:18:31.119393 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.119352 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrq6x"] Jul 07 02:18:31.137032 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.137002 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9g6n5"] Jul 07 02:18:31.137189 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.137136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.140199 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.140164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pm56q\"" Jul 07 02:18:31.140398 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.140223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 02:18:31.140398 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.140164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 02:18:31.143523 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.142850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 02:18:31.156809 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.156745 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrq6x"] Jul 07 02:18:31.156809 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.156801 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9g6n5"] Jul 07 02:18:31.157010 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.156893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.167444 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.167419 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\"" Jul 07 02:18:31.167755 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.167741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 02:18:31.167845 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.167825 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 02:18:31.217064 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.217024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948fc827-33f2-4c0b-96d8-229300467070-cert\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.217064 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.217069 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/948fc827-33f2-4c0b-96d8-229300467070-kube-api-access-4v5sj\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.317853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzc5\" (UniqueName: \"kubernetes.io/projected/44d6268e-27d4-40eb-b265-ff0ddd3475d6-kube-api-access-smzc5\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.317853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948fc827-33f2-4c0b-96d8-229300467070-cert\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.317853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d6268e-27d4-40eb-b265-ff0ddd3475d6-tmp-dir\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.318461 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/948fc827-33f2-4c0b-96d8-229300467070-kube-api-access-4v5sj\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.318461 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d6268e-27d4-40eb-b265-ff0ddd3475d6-config-volume\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.318461 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.317914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.322686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.322653 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948fc827-33f2-4c0b-96d8-229300467070-cert\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.330632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.330600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/948fc827-33f2-4c0b-96d8-229300467070-kube-api-access-4v5sj\") pod \"ingress-canary-mrq6x\" (UID: \"948fc827-33f2-4c0b-96d8-229300467070\") " pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.345065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.345020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.419160 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.418948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d6268e-27d4-40eb-b265-ff0ddd3475d6-config-volume\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.419352 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.419178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.419352 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.419227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzc5\" (UniqueName: \"kubernetes.io/projected/44d6268e-27d4-40eb-b265-ff0ddd3475d6-kube-api-access-smzc5\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.419352 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.419251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d6268e-27d4-40eb-b265-ff0ddd3475d6-tmp-dir\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.419352 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:31.419291 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:31.419549 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:31.419380 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:31.919356741 +0000 UTC m=+36.676905749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:31.419549 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.419519 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d6268e-27d4-40eb-b265-ff0ddd3475d6-tmp-dir\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.419619 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.419575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44d6268e-27d4-40eb-b265-ff0ddd3475d6-config-volume\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.429623 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.429590 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzc5\" (UniqueName: \"kubernetes.io/projected/44d6268e-27d4-40eb-b265-ff0ddd3475d6-kube-api-access-smzc5\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.438466 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.438416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:31.450315 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.450282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrq6x" Jul 07 02:18:31.650535 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.650497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrq6x"] Jul 07 02:18:31.655300 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:18:31.655254 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948fc827_33f2_4c0b_96d8_229300467070.slice/crio-673b6cadc8571ab061ec515d0b378112094c3b73254345f90acaec40eda26185 WatchSource:0}: Error finding container 673b6cadc8571ab061ec515d0b378112094c3b73254345f90acaec40eda26185: Status 404 returned error can't find the container with id 673b6cadc8571ab061ec515d0b378112094c3b73254345f90acaec40eda26185 Jul 07 02:18:31.923550 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:31.923461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:31.923702 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:31.923626 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:31.923702 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:31.923694 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:32.923677024 +0000 UTC m=+37.681226031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:32.000063 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.000031 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3bc19ab-4f53-46d0-958e-52ff15ae512e" containerID="3cafc344c341400e49451ca3d948bf3916fa1ca97d743f79b3674825b08ec178" exitCode=0 Jul 07 02:18:32.000241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.000109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerDied","Data":"3cafc344c341400e49451ca3d948bf3916fa1ca97d743f79b3674825b08ec178"} Jul 07 02:18:32.001439 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.001414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrq6x" event={"ID":"948fc827-33f2-4c0b-96d8-229300467070","Type":"ContainerStarted","Data":"673b6cadc8571ab061ec515d0b378112094c3b73254345f90acaec40eda26185"} Jul 07 02:18:32.786219 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.786178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:18:32.787020 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.786184 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:18:32.787020 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.786322 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:18:32.790366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790339 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 02:18:32.790366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\"" Jul 07 02:18:32.790597 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\"" Jul 07 02:18:32.790597 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 02:18:32.790597 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 02:18:32.790806 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.790771 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 02:18:32.932601 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:32.932559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:32.932822 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:32.932802 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:32.932887 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:32.932877 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:34.932854768 +0000 UTC m=+39.690403774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:33.009175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:33.009136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" event={"ID":"e3bc19ab-4f53-46d0-958e-52ff15ae512e","Type":"ContainerStarted","Data":"145780ddcfb271d1531056d52e30e4c3aa09e89b6ffb6363879835653bce9cd6"} Jul 07 02:18:33.036576 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:33.036452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5zxmf" podStartSLOduration=8.437535248 podStartE2EDuration="38.03643549s" totalImagesPullingTime="29.598900242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:33.03472047 +0000 UTC m=+37.792269498" watchObservedRunningTime="2026-07-07 02:18:33.03643549 +0000 UTC m=+37.793984514" Jul 07 02:18:33.132964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:33.132931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:33.198391 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:33.198333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:34.950605 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:34.950373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:34.951195 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:34.950556 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:34.951195 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:34.950696 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:38.950677931 +0000 UTC m=+43.708226935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:35.015717 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:35.015633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrq6x" event={"ID":"948fc827-33f2-4c0b-96d8-229300467070","Type":"ContainerStarted","Data":"6c5888787d0945613dba153fbb1909ecd29ea3d23b97a551ba021ead7483c9ca"} Jul 07 02:18:35.034405 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:35.034345 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrq6x" podStartSLOduration=1.5495392510000001 podStartE2EDuration="4.034326881s" totalImagesPullingTime="2.48478763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:35.03357328 +0000 UTC m=+39.791122305" watchObservedRunningTime="2026-07-07 02:18:35.034326881 +0000 UTC m=+39.791875905" Jul 07 02:18:36.176370 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:36.176298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:36.238673 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:36.238639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:38.978469 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:38.978431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:38.978897 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:38.978547 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:38.978897 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:38.978616 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:46.978600983 +0000 UTC m=+51.736149986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:42.766037 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:42.765993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.270994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:46.270926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.334103 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:46.334065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.393915 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:46.393871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:46.908363 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:46.908317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.009262 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:47.009222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.039215 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:47.039175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:18:47.039408 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:47.039328 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:18:47.039408 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:18:47.039396 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:03.039377424 +0000 UTC m=+67.796926427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:18:47.376715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:47.376673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:47.435422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:47.435362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:56.991962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:56.991928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qz6ck" Jul 07 02:18:57.203000 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:57.202958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:18:57.282066 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:18:57.282036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:01.536927 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.536880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:19:01.537369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.536941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:19:01.537369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.537007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:19:01.540182 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.540155 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 02:19:01.540290 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.540242 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 02:19:01.540351 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.540301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 02:19:01.547359 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:01.547318 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 02:19:01.547596 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:01.547394 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:20:05.54737583 +0000 UTC m=+130.304924838 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : secret "metrics-daemon-secret" not found Jul 07 02:19:01.549994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.549975 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 02:19:01.551275 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.551246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40706701-1bf7-40be-b011-fb80c37fe4c1-original-pull-secret\") pod \"global-pull-secret-syncer-zj6zc\" (UID: \"40706701-1bf7-40be-b011-fb80c37fe4c1\") " pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:19:01.560768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.560733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7nq\" (UniqueName: \"kubernetes.io/projected/739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70-kube-api-access-mt7nq\") pod \"network-check-target-f85jb\" (UID: \"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70\") " pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:19:01.598742 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.598701 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zj6zc" Jul 07 02:19:01.617994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.617960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\"" Jul 07 02:19:01.625238 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.625207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:19:01.774636 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.774599 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zj6zc"] Jul 07 02:19:01.777699 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:19:01.777664 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40706701_1bf7_40be_b011_fb80c37fe4c1.slice/crio-99bc20edff8593014000fc4a3dca8a90a9d9340eb908bcc72686873b92d3e031 WatchSource:0}: Error finding container 99bc20edff8593014000fc4a3dca8a90a9d9340eb908bcc72686873b92d3e031: Status 404 returned error can't find the container with id 99bc20edff8593014000fc4a3dca8a90a9d9340eb908bcc72686873b92d3e031 Jul 07 02:19:01.790086 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:01.790008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f85jb"] Jul 07 02:19:01.802481 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:19:01.802446 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739c2f9f_3d3e_41e6_a6fb_2ba46caf9f70.slice/crio-6ab9207c161df6c43a83878e30acf8e8d322acd133cb7afe778f91e4e15cfbba WatchSource:0}: Error finding container 6ab9207c161df6c43a83878e30acf8e8d322acd133cb7afe778f91e4e15cfbba: Status 404 returned error can't find the container with id 6ab9207c161df6c43a83878e30acf8e8d322acd133cb7afe778f91e4e15cfbba Jul 07 02:19:02.082993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:02.082892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f85jb" event={"ID":"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70","Type":"ContainerStarted","Data":"6ab9207c161df6c43a83878e30acf8e8d322acd133cb7afe778f91e4e15cfbba"} Jul 07 02:19:02.084110 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:02.084079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zj6zc" event={"ID":"40706701-1bf7-40be-b011-fb80c37fe4c1","Type":"ContainerStarted","Data":"99bc20edff8593014000fc4a3dca8a90a9d9340eb908bcc72686873b92d3e031"} Jul 07 02:19:03.047085 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:03.047041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:19:03.047630 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:03.047225 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:19:03.047630 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:03.047295 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:35.047275162 +0000 UTC m=+99.804824169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:19:07.101027 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:07.100990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f85jb" event={"ID":"739c2f9f-3d3e-41e6-a6fb-2ba46caf9f70","Type":"ContainerStarted","Data":"d1b5e60f244f971489aed731a8938e0542e0d74915ffcc3d2e1a149b0dc887fc"} Jul 07 02:19:07.101503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:07.101122 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:19:07.102601 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:07.102580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zj6zc" event={"ID":"40706701-1bf7-40be-b011-fb80c37fe4c1","Type":"ContainerStarted","Data":"8c517410eceeace70277b570f5aa09ac940571d85f4371485c14ce512a0293ca"} Jul 07 02:19:07.118085 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:07.118027 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f85jb" podStartSLOduration=67.581053523 podStartE2EDuration="1m12.118012483s" totalImagesPullingTime="4.53695896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:07.117974151 +0000 UTC m=+71.875523177" watchObservedRunningTime="2026-07-07 02:19:07.118012483 +0000 UTC m=+71.875561486" Jul 07 02:19:07.136200 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:07.136144 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zj6zc" podStartSLOduration=67.569286907 podStartE2EDuration="1m12.136126582s" totalImagesPullingTime="4.566839675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:07.135688327 +0000 UTC m=+71.893237380" watchObservedRunningTime="2026-07-07 02:19:07.136126582 +0000 UTC m=+71.893675606" Jul 07 02:19:08.304752 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:08.304708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:08.375212 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:08.375174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:19:11.124704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.124665 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld"] Jul 07 02:19:11.126881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.126863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.129812 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.129770 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 02:19:11.130364 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.130336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 02:19:11.133817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.131589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 02:19:11.133817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.131888 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 02:19:11.133817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.132168 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 02:19:11.133817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.132509 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 02:19:11.133817 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.132533 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 02:19:11.142701 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.142667 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld"] Jul 07 02:19:11.202375 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.202375 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.202613 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ld9\" (UniqueName: \"kubernetes.io/projected/a25cfe3c-c992-46d5-a7e0-a24780df4e54-kube-api-access-b6ld9\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.202613 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.202613 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25cfe3c-c992-46d5-a7e0-a24780df4e54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.202613 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.202571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303289 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ld9\" (UniqueName: \"kubernetes.io/projected/a25cfe3c-c992-46d5-a7e0-a24780df4e54-kube-api-access-b6ld9\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303289 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25cfe3c-c992-46d5-a7e0-a24780df4e54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.303496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.303399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.304157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.304128 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25cfe3c-c992-46d5-a7e0-a24780df4e54-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.306146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.306122 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.306247 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.306143 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.306333 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.306315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-ca\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.306446 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.306430 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a25cfe3c-c992-46d5-a7e0-a24780df4e54-hub\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.313318 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.313292 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ld9\" (UniqueName: \"kubernetes.io/projected/a25cfe3c-c992-46d5-a7e0-a24780df4e54-kube-api-access-b6ld9\") pod \"cluster-proxy-proxy-agent-6b7979bd9-flcld\" (UID: \"a25cfe3c-c992-46d5-a7e0-a24780df4e54\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.449267 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.449166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:19:11.586218 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:11.586185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld"] Jul 07 02:19:11.589565 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:19:11.589524 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25cfe3c_c992_46d5_a7e0_a24780df4e54.slice/crio-a0bd12a4eea67e9017eb319035f27af40892a4a981a67b15603ec9170fe93630 WatchSource:0}: Error finding container a0bd12a4eea67e9017eb319035f27af40892a4a981a67b15603ec9170fe93630: Status 404 returned error can't find the container with id a0bd12a4eea67e9017eb319035f27af40892a4a981a67b15603ec9170fe93630 Jul 07 02:19:12.116196 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:12.116155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerStarted","Data":"a0bd12a4eea67e9017eb319035f27af40892a4a981a67b15603ec9170fe93630"} Jul 07 02:19:17.131539 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:17.131499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerStarted","Data":"5c324caff6f5d85738f661c557f2382f933767e46b38ccf7ca2bfe42a2d37d7a"} Jul 07 02:19:17.132070 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:17.131547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerStarted","Data":"d1dd1a66fcb3dc89a205694543df0efc376c3076265cd3c1d3986987643a7549"} Jul 07 02:19:17.132070 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:17.131565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerStarted","Data":"d544fbf29ad316b5a5e8640e8271fb4c1b5a5befd0bace2ee1dbd9ba4e458041"} Jul 07 02:19:17.150929 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:17.150872 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" podStartSLOduration=1.51770559 podStartE2EDuration="6.150857305s" totalImagesPullingTime="4.633151715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:17.149664786 +0000 UTC m=+81.907213805" watchObservedRunningTime="2026-07-07 02:19:17.150857305 +0000 UTC m=+81.908406330" Jul 07 02:19:35.077810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:35.077745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:19:35.078262 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:35.077902 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:19:35.078262 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:19:35.077971 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:20:39.07795325 +0000 UTC m=+163.835502253 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:19:38.107323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:19:38.107293 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f85jb" Jul 07 02:20:05.599111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:20:05.599062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:20:05.599608 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:05.599223 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 02:20:05.599608 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:05.599302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.599285257 +0000 UTC m=+252.356834260 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : secret "metrics-daemon-secret" not found Jul 07 02:20:34.166996 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:34.166941 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9g6n5" podUID="44d6268e-27d4-40eb-b265-ff0ddd3475d6" Jul 07 02:20:34.355156 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:20:34.355118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:20:35.806372 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:35.806316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:20:36.831542 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:20:36.831504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:20:39.135458 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:20:39.135420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:20:39.135851 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:39.135524 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 02:20:39.135851 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:20:39.135583 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls podName:44d6268e-27d4-40eb-b265-ff0ddd3475d6 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:41.135568196 +0000 UTC m=+285.893117198 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls") pod "dns-default-9g6n5" (UID: "44d6268e-27d4-40eb-b265-ff0ddd3475d6") : secret "dns-default-metrics-tls" not found Jul 07 02:20:50.785563 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:20:50.785479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:21:34.309027 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:21:34.308979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:05.495997 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.495954 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:05.498948 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.498923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.502123 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.502103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 02:22:05.502692 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.502674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkr4v\"" Jul 07 02:22:05.503697 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.503673 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 02:22:05.503697 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.503692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 02:22:05.507751 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.507727 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 02:22:05.517961 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.517924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:05.622173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622389 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622389 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622389 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622483 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622483 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5wf\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.622483 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.622429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.634481 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.634446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:05.723059 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723059 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5wf\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723566 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723566 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:05.723533 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:22:05.723566 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:05.723555 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd78fc48-gdj65: secret "image-registry-tls" not found Jul 07 02:22:05.723715 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:05.723625 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls podName:ca6069cb-1eaa-41a3-818d-42c744eb74a9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:06.223605969 +0000 UTC m=+250.981154974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls") pod "image-registry-5dd78fc48-gdj65" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9") : secret "image-registry-tls" not found Jul 07 02:22:05.723715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723691 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.723917 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.723896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.724337 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.724321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.725856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.725837 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.726015 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.725994 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.731918 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.731884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:05.733064 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.733042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5wf\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:05.733178 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:05.733156 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:06.227032 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:06.226990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:06.227262 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:06.227153 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:22:06.227262 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:06.227179 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd78fc48-gdj65: secret "image-registry-tls" not found Jul 07 02:22:06.227262 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:06.227260 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls podName:ca6069cb-1eaa-41a3-818d-42c744eb74a9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.227237184 +0000 UTC m=+251.984786188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls") pod "image-registry-5dd78fc48-gdj65" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9") : secret "image-registry-tls" not found Jul 07 02:22:07.233736 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:07.233693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:07.234154 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:07.233841 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:22:07.234154 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:07.233855 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd78fc48-gdj65: secret "image-registry-tls" not found Jul 07 02:22:07.234154 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:07.233917 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls podName:ca6069cb-1eaa-41a3-818d-42c744eb74a9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:09.233902212 +0000 UTC m=+253.991451215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls") pod "image-registry-5dd78fc48-gdj65" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9") : secret "image-registry-tls" not found Jul 07 02:22:07.636337 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:07.636292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:22:07.636508 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:07.636441 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 02:22:07.636508 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:07.636507 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs podName:b7b9e554-ea7a-4647-b208-0cbb59badeb6 nodeName:}" failed. No retries permitted until 2026-07-07 02:24:09.63649125 +0000 UTC m=+374.394040253 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs") pod "network-metrics-daemon-t4vsh" (UID: "b7b9e554-ea7a-4647-b208-0cbb59badeb6") : secret "metrics-daemon-secret" not found Jul 07 02:22:09.246659 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:09.246606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:09.247113 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:09.246777 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:22:09.247113 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:09.246830 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd78fc48-gdj65: secret "image-registry-tls" not found Jul 07 02:22:09.247113 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:09.246904 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls podName:ca6069cb-1eaa-41a3-818d-42c744eb74a9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:13.24688403 +0000 UTC m=+258.004433052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls") pod "image-registry-5dd78fc48-gdj65" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9") : secret "image-registry-tls" not found Jul 07 02:22:09.728226 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:09.728175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:09.956137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:09.956097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:10.181401 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.181360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:10.240491 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.240463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:10.451946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.451852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:10.477603 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.477569 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzghs"] Jul 07 02:22:10.481883 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.481859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.484720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.484692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 02:22:10.485806 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.485761 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 02:22:10.486503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.486455 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hsdg\"" Jul 07 02:22:10.486643 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.486541 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 02:22:10.486643 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.486468 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 02:22:10.497151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.497116 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzghs"] Jul 07 02:22:10.545413 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.545377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:10.558604 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.558562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-data-volume\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.558816 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.558646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.558816 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.558688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-crio-socket\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.558816 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.558708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.558952 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.558817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wgm\" (UniqueName: \"kubernetes.io/projected/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-api-access-z5wgm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659675 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.659630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-data-volume\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659675 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.659674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.659698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-crio-socket\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.659717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:10.659774 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.659847 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-crio-socket\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:10.659859 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:22:10.659946 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:10.659867 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.159850825 +0000 UTC m=+255.917399827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : configmap "kube-rbac-proxy" not found Jul 07 02:22:10.660173 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:10.659965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.159949853 +0000 UTC m=+255.917498859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : secret "insights-runtime-extractor-tls" not found Jul 07 02:22:10.660173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.660011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wgm\" (UniqueName: \"kubernetes.io/projected/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-api-access-z5wgm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.660173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.660066 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-data-volume\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:10.668906 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:10.668873 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wgm\" (UniqueName: \"kubernetes.io/projected/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-api-access-z5wgm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:11.163970 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:11.163929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:11.163970 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:11.163969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:11.164313 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:11.164082 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:22:11.164313 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:11.164118 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:22:11.164313 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:11.164158 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.164142745 +0000 UTC m=+256.921691748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : secret "insights-runtime-extractor-tls" not found Jul 07 02:22:11.164313 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:11.164173 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.16416646 +0000 UTC m=+256.921715463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : configmap "kube-rbac-proxy" not found Jul 07 02:22:11.701472 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:11.701436 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:11.798230 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:11.798190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:11.899141 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:11.899107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:12.084395 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:12.084350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:12.172208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:12.172163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:12.172208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:12.172204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:12.172488 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:12.172354 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:22:12.172488 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:12.172361 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:22:12.172488 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:12.172418 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.172400573 +0000 UTC m=+258.929949597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : configmap "kube-rbac-proxy" not found Jul 07 02:22:12.172488 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:12.172444 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.172425607 +0000 UTC m=+258.929974625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : secret "insights-runtime-extractor-tls" not found Jul 07 02:22:12.464819 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:12.464726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq4gw_a946adcf-84da-4601-a697-821c2c425a08/dns-node-resolver/0.log" Jul 07 02:22:12.980425 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:12.980380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:13.280191 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:13.280076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:13.280360 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:13.280249 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 02:22:13.280360 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:13.280270 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd78fc48-gdj65: secret "image-registry-tls" not found Jul 07 02:22:13.280360 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:13.280331 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls podName:ca6069cb-1eaa-41a3-818d-42c744eb74a9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:21.280314618 +0000 UTC m=+266.037863623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls") pod "image-registry-5dd78fc48-gdj65" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9") : secret "image-registry-tls" not found Jul 07 02:22:13.465591 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:13.465554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j96jg_b434ae8a-907e-4260-aa20-16ba66f6e467/node-ca/0.log" Jul 07 02:22:14.186845 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:14.186777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:14.186845 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:14.186842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:14.187271 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:14.186919 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:22:14.187271 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:14.186929 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:22:14.187271 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:14.186987 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.186969298 +0000 UTC m=+262.944518300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : secret "insights-runtime-extractor-tls" not found Jul 07 02:22:14.187271 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:14.187002 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.186995232 +0000 UTC m=+262.944544234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : configmap "kube-rbac-proxy" not found Jul 07 02:22:14.265005 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:14.264972 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrq6x_948fc827-33f2-4c0b-96d8-229300467070/serve-healthcheck-canary/0.log" Jul 07 02:22:15.522540 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:15.522505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:15.699874 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:15.699829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:15.796868 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:15.796831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:15.897508 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:15.897420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:15.998151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:15.998115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:16.113230 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:16.113132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:16.428920 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:16.428826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:17.232990 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:17.232959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:17.431937 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:17.431900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:17.629416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:17.629378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:18.025666 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:18.025570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:18.219388 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:18.219347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:18.219388 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:18.219393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:18.219637 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:18.219480 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 02:22:18.219637 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:18.219521 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 02:22:18.219637 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:18.219604 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.219583491 +0000 UTC m=+270.977132497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : secret "insights-runtime-extractor-tls" not found Jul 07 02:22:18.219637 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:18.219622 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm podName:b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.219615249 +0000 UTC m=+270.977164252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzghs" (UID: "b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e") : configmap "kube-rbac-proxy" not found Jul 07 02:22:18.821678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:18.821636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:19.227966 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:19.227863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:19.284266 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:19.284232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:19.428574 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:19.428530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:19.492688 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:19.492585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:19.632190 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:19.632150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:20.021592 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:20.021556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:20.625871 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:20.625832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:21.031419 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.031319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:21.257389 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.257337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:21.343325 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.343286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:21.346037 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.345999 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"image-registry-5dd78fc48-gdj65\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:21.409267 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.409211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:21.558315 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.558278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:21.561995 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:21.561963 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6069cb_1eaa_41a3_818d_42c744eb74a9.slice/crio-fe02e1dff2803167a0388e3999bc4b0a489ff805496b31688cb1a925a7951bc8 WatchSource:0}: Error finding container fe02e1dff2803167a0388e3999bc4b0a489ff805496b31688cb1a925a7951bc8: Status 404 returned error can't find the container with id fe02e1dff2803167a0388e3999bc4b0a489ff805496b31688cb1a925a7951bc8 Jul 07 02:22:21.667922 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.667882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" event={"ID":"ca6069cb-1eaa-41a3-818d-42c744eb74a9","Type":"ContainerStarted","Data":"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5"} Jul 07 02:22:21.668176 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.667936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" event={"ID":"ca6069cb-1eaa-41a3-818d-42c744eb74a9","Type":"ContainerStarted","Data":"fe02e1dff2803167a0388e3999bc4b0a489ff805496b31688cb1a925a7951bc8"} Jul 07 02:22:21.668176 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.668017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:21.688402 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.688338 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" podStartSLOduration=16.688318802 podStartE2EDuration="16.688318802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:21.687107429 +0000 UTC m=+266.444656454" watchObservedRunningTime="2026-07-07 02:22:21.688318802 +0000 UTC m=+266.445867826" Jul 07 02:22:21.828558 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:21.828519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:22.237468 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:22.237433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:22.293364 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:22.293324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:22.627140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:22.627095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:22.825057 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:22.825021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:23.229978 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:23.229937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:23.371979 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:23.371939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:23.827587 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:23.827549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:24.230455 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:24.230368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:24.433338 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:24.433304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:24.687223 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:24.687186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:24.827902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:24.827870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:25.222962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:25.222925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:25.836035 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:25.835987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:26.030647 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.030610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:26.228084 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.227980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:26.285668 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.285628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:26.285668 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.285669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:26.286205 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.286186 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:26.288375 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.288349 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzghs\" (UID: \"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e\") " pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:26.392893 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.392850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzghs" Jul 07 02:22:26.506570 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.506534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:26.546472 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.546441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzghs"] Jul 07 02:22:26.549164 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:26.549126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e6b0b5_ad25_4afe_ad0e_431e3ba9475e.slice/crio-0b1bf88b5e7efd7c133533ada999a2cd713b11fdb98434327141b08696970690 WatchSource:0}: Error finding container 0b1bf88b5e7efd7c133533ada999a2cd713b11fdb98434327141b08696970690: Status 404 returned error can't find the container with id 0b1bf88b5e7efd7c133533ada999a2cd713b11fdb98434327141b08696970690 Jul 07 02:22:26.601996 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.601948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:26.683325 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:26.683287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzghs" event={"ID":"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e","Type":"ContainerStarted","Data":"0b1bf88b5e7efd7c133533ada999a2cd713b11fdb98434327141b08696970690"} Jul 07 02:22:30.700460 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:30.700419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzghs" event={"ID":"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e","Type":"ContainerStarted","Data":"0a686c4e2d6575a21c423fb3bae8dbcc1bebd9a120864689e75e0ce06ad9b779"} Jul 07 02:22:30.827192 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:30.827150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:30.935435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:30.935383 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:30.965380 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:30.965340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.098566 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.098457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.200120 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.200077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.302327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.302283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.633202 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.633162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.700177 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.700140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.705253 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.705226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzghs" event={"ID":"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e","Type":"ContainerStarted","Data":"79e40cab1bdb64dc4226b43a96945f933abfc6fbe649fc8639d048cb2249e887"} Jul 07 02:22:31.818089 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.818030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:31.909410 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:31.909292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:32.423619 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:32.423577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:32.637941 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:32.637899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:33.012091 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:33.012044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:33.112084 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:33.112046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:33.172121 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:33.172082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:34.718072 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:34.718029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzghs" event={"ID":"b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e","Type":"ContainerStarted","Data":"f605c6a84c99c624362c171db481238c1516182ff585e499c0a933756abbdd69"} Jul 07 02:22:34.735663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:34.735601 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzghs" podStartSLOduration=17.505563152 podStartE2EDuration="24.735579719s" totalImagesPullingTime="7.230016567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:34.73516719 +0000 UTC m=+279.492716231" watchObservedRunningTime="2026-07-07 02:22:34.735579719 +0000 UTC m=+279.493128742" Jul 07 02:22:37.356672 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:37.356617 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9g6n5" podUID="44d6268e-27d4-40eb-b265-ff0ddd3475d6" Jul 07 02:22:37.728449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:37.728368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:40.166897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.166857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:40.267111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.267080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:40.340973 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.340928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:40.506733 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.506633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:40.941752 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.941718 2564 patch_prober.go:28] interesting pod/image-registry-5dd78fc48-gdj65 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 02:22:40.941963 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:40.941772 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:22:41.133233 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.133198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:41.198514 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.198428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:41.204710 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.204674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:41.207223 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.207195 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44d6268e-27d4-40eb-b265-ff0ddd3475d6-metrics-tls\") pod \"dns-default-9g6n5\" (UID: \"44d6268e-27d4-40eb-b265-ff0ddd3475d6\") " pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:41.331799 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.331761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\"" Jul 07 02:22:41.339966 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.339934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:41.469355 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.469281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9g6n5"] Jul 07 02:22:41.473256 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:41.473229 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d6268e_27d4_40eb_b265_ff0ddd3475d6.slice/crio-0dfb39d53cfeb2d8937b6ef4d7c49008915797ae85d855aa9954482e63171fc9 WatchSource:0}: Error finding container 0dfb39d53cfeb2d8937b6ef4d7c49008915797ae85d855aa9954482e63171fc9: Status 404 returned error can't find the container with id 0dfb39d53cfeb2d8937b6ef4d7c49008915797ae85d855aa9954482e63171fc9 Jul 07 02:22:41.741686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:41.741594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9g6n5" event={"ID":"44d6268e-27d4-40eb-b265-ff0ddd3475d6","Type":"ContainerStarted","Data":"0dfb39d53cfeb2d8937b6ef4d7c49008915797ae85d855aa9954482e63171fc9"} Jul 07 02:22:42.853656 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.853620 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:22:42.856272 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.856251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:42.862102 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.862080 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 02:22:42.863171 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863141 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bvcr9\"" Jul 07 02:22:42.863171 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863153 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 02:22:42.863336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863145 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 02:22:42.863336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863194 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 02:22:42.863336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863291 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 02:22:42.863489 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 02:22:42.863529 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.863485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 02:22:42.867881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:42.867862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:22:43.017084 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.017084 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.017323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkg6\" (UniqueName: \"kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.017323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.017323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.017323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.017295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118262 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118262 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118262 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkg6\" (UniqueName: \"kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118531 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118531 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.118531 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.118346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.119254 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.119042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.119254 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.119042 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.119254 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.119208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.121293 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.121239 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.121449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.121424 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.127706 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.127675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkg6\" (UniqueName: \"kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6\") pod \"console-8564dc94c8-psh89\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.165469 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.165435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:43.488658 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.488637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:22:43.498819 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:43.498766 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5edbcfb_6d24_409b_b183_a8e4d92af5a2.slice/crio-ef34bee0d70d1a7a41d456c78e32fe9a2ae5efcba40daa9c087a029799d6b15a WatchSource:0}: Error finding container ef34bee0d70d1a7a41d456c78e32fe9a2ae5efcba40daa9c087a029799d6b15a: Status 404 returned error can't find the container with id ef34bee0d70d1a7a41d456c78e32fe9a2ae5efcba40daa9c087a029799d6b15a Jul 07 02:22:43.750294 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.750201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9g6n5" event={"ID":"44d6268e-27d4-40eb-b265-ff0ddd3475d6","Type":"ContainerStarted","Data":"174885b7dae2aed77c75524200a0befbe1cd293807405e68bd3aa63d4eece4f8"} Jul 07 02:22:43.750294 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.750256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9g6n5" event={"ID":"44d6268e-27d4-40eb-b265-ff0ddd3475d6","Type":"ContainerStarted","Data":"48eaeaf219002b5816d100fb0371057268ed6403c4969f53e247f6b7f40d0c6d"} Jul 07 02:22:43.750502 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.750309 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:43.751539 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.751516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564dc94c8-psh89" event={"ID":"c5edbcfb-6d24-409b-b183-a8e4d92af5a2","Type":"ContainerStarted","Data":"ef34bee0d70d1a7a41d456c78e32fe9a2ae5efcba40daa9c087a029799d6b15a"} Jul 07 02:22:43.771749 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:43.771699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9g6n5" podStartSLOduration=250.843585038 podStartE2EDuration="4m12.771682031s" totalImagesPullingTime="1.928096993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:43.771076498 +0000 UTC m=+288.528625522" watchObservedRunningTime="2026-07-07 02:22:43.771682031 +0000 UTC m=+288.529231055" Jul 07 02:22:46.764982 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:46.764947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564dc94c8-psh89" event={"ID":"c5edbcfb-6d24-409b-b183-a8e4d92af5a2","Type":"ContainerStarted","Data":"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c"} Jul 07 02:22:46.782892 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:46.782841 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8564dc94c8-psh89" podStartSLOduration=1.752028969 podStartE2EDuration="4.782825041s" totalImagesPullingTime="3.030796072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:46.782228216 +0000 UTC m=+291.539777241" watchObservedRunningTime="2026-07-07 02:22:46.782825041 +0000 UTC m=+291.540374065" Jul 07 02:22:47.059094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:47.058983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:48.268742 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:48.268706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:49.449758 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.449726 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:22:49.452676 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.452659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.463202 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.463181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 02:22:49.465221 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.465198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:22:49.562944 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.562904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdr8\" (UniqueName: \"kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.562944 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.562947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.563151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.562975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.563151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.563064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.563151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.563095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.563151 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.563128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.563273 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.563181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.650387 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.650355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:49.664314 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdr8\" (UniqueName: \"kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.664586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.664461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.665263 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.665200 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.665263 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.665222 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.665417 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.665312 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.665468 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.665444 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.667131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.667108 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.667190 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.667180 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.675136 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.675115 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdr8\" (UniqueName: \"kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8\") pod \"console-84c6494674-sp8mh\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.753568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.753471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:49.761585 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.761560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:49.894878 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:49.894845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:22:49.897701 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:49.897674 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6f6aee_cd9d_4d4f_a4bc_5bb85ec9261a.slice/crio-49bd1b096426ae96ee830927b5c0858d3582f10392d04c9ea93b8a67c2ee8fcc WatchSource:0}: Error finding container 49bd1b096426ae96ee830927b5c0858d3582f10392d04c9ea93b8a67c2ee8fcc: Status 404 returned error can't find the container with id 49bd1b096426ae96ee830927b5c0858d3582f10392d04c9ea93b8a67c2ee8fcc Jul 07 02:22:50.252850 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.252816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:50.355970 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.355942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:50.418474 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.418442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:50.783812 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.783763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6494674-sp8mh" event={"ID":"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a","Type":"ContainerStarted","Data":"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1"} Jul 07 02:22:50.784254 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.783852 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6494674-sp8mh" event={"ID":"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a","Type":"ContainerStarted","Data":"49bd1b096426ae96ee830927b5c0858d3582f10392d04c9ea93b8a67c2ee8fcc"} Jul 07 02:22:50.808094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.808043 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c6494674-sp8mh" podStartSLOduration=1.808018474 podStartE2EDuration="1.808018474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:50.806293505 +0000 UTC m=+295.563842531" watchObservedRunningTime="2026-07-07 02:22:50.808018474 +0000 UTC m=+295.565567499" Jul 07 02:22:50.940450 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.940404 2564 patch_prober.go:28] interesting pod/image-registry-5dd78fc48-gdj65 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 02:22:50.940605 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:50.940473 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:22:51.361411 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.361376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:51.400478 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.400445 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m"] Jul 07 02:22:51.403663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.403645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.406558 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.406537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 02:22:51.406662 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.406567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 02:22:51.406662 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.406571 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 02:22:51.407878 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.407834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 02:22:51.407878 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.407853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t767j\"" Jul 07 02:22:51.407878 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.407834 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 02:22:51.414772 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.414754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m"] Jul 07 02:22:51.436863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.436828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:51.579372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.579337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.579372 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.579376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6755d51e-edde-4e70-a628-513836117e82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.579583 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.579406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fws\" (UniqueName: \"kubernetes.io/projected/6755d51e-edde-4e70-a628-513836117e82-kube-api-access-m4fws\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.579583 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.579514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.680912 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.680816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6755d51e-edde-4e70-a628-513836117e82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.680912 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.680900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fws\" (UniqueName: \"kubernetes.io/projected/6755d51e-edde-4e70-a628-513836117e82-kube-api-access-m4fws\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.681080 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.680944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.681080 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.681007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.681163 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:51.681097 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 02:22:51.681209 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:51.681170 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls podName:6755d51e-edde-4e70-a628-513836117e82 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:52.181151485 +0000 UTC m=+296.938700488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-vqw2m" (UID: "6755d51e-edde-4e70-a628-513836117e82") : secret "prometheus-operator-tls" not found Jul 07 02:22:51.681514 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.681494 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6755d51e-edde-4e70-a628-513836117e82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.683778 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.683754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:51.690033 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:51.690011 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fws\" (UniqueName: \"kubernetes.io/projected/6755d51e-edde-4e70-a628-513836117e82-kube-api-access-m4fws\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:52.079102 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.079048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:52.179597 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.179541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:52.184797 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.184765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:52.187309 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.187289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6755d51e-edde-4e70-a628-513836117e82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vqw2m\" (UID: \"6755d51e-edde-4e70-a628-513836117e82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:52.253898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.253857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:52.312452 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.312416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" Jul 07 02:22:52.448398 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.448368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m"] Jul 07 02:22:52.451191 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:52.451161 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6755d51e_edde_4e70_a628_513836117e82.slice/crio-4f5f8b191cecd12779e3125340cf7f52f7e66ad74c1844e46ac7414c324002db WatchSource:0}: Error finding container 4f5f8b191cecd12779e3125340cf7f52f7e66ad74c1844e46ac7414c324002db: Status 404 returned error can't find the container with id 4f5f8b191cecd12779e3125340cf7f52f7e66ad74c1844e46ac7414c324002db Jul 07 02:22:52.792759 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:52.792721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" event={"ID":"6755d51e-edde-4e70-a628-513836117e82","Type":"ContainerStarted","Data":"4f5f8b191cecd12779e3125340cf7f52f7e66ad74c1844e46ac7414c324002db"} Jul 07 02:22:53.165984 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.165884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:53.165984 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.165957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:53.170753 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.170699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.171742 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.171721 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:53.758288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.758261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9g6n5" Jul 07 02:22:53.786277 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:53.786240 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t4vsh" podUID="b7b9e554-ea7a-4647-b208-0cbb59badeb6" Jul 07 02:22:53.800419 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.800390 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:22:53.886650 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.886620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:53.976118 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:53.975995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.077319 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.077268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.177391 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.177352 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.238627 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.238597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:54.800512 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.800477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" event={"ID":"6755d51e-edde-4e70-a628-513836117e82","Type":"ContainerStarted","Data":"066f5fb8c9b987f73f18a82e2fc3a60a7f24da1d44de516f05c04fdd9348fe4f"} Jul 07 02:22:54.800687 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.800531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" event={"ID":"6755d51e-edde-4e70-a628-513836117e82","Type":"ContainerStarted","Data":"96992c01bcc622c18585f0fd8318eb7b550874f0061ebbb115201b6127f9062a"} Jul 07 02:22:54.818693 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:54.818638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vqw2m" podStartSLOduration=2.442260222 podStartE2EDuration="3.818620566s" totalImagesPullingTime="1.376360344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:54.817669589 +0000 UTC m=+299.575218613" watchObservedRunningTime="2026-07-07 02:22:54.818620566 +0000 UTC m=+299.576169591" Jul 07 02:22:55.412807 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:55.412756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:55.650007 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:55.649981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:22:55.650351 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:55.650334 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:22:55.657264 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:55.657244 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 02:22:55.957092 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:55.956937 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" containerID="cri-o://d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5" gracePeriod=30 Jul 07 02:22:56.206822 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.206770 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:56.216016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.215943 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.215985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216033 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn5wf\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216089 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216157 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216183 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted\") pod \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\" (UID: \"ca6069cb-1eaa-41a3-818d-42c744eb74a9\") " Jul 07 02:22:56.216493 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216459 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:22:56.216641 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.216610 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:22:56.218830 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.218769 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:22:56.218830 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.218812 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:22:56.219023 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.218919 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf" (OuterVolumeSpecName: "kube-api-access-wn5wf") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "kube-api-access-wn5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:22:56.219023 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.218939 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:22:56.219023 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.218981 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:22:56.225470 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.225431 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ca6069cb-1eaa-41a3-818d-42c744eb74a9" (UID: "ca6069cb-1eaa-41a3-818d-42c744eb74a9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:22:56.317429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317395 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-tls\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317425 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5wf\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-kube-api-access-wn5wf\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317436 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-trusted-ca\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317444 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-installation-pull-secrets\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317702 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317454 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6069cb-1eaa-41a3-818d-42c744eb74a9-bound-sa-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317702 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317463 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6069cb-1eaa-41a3-818d-42c744eb74a9-ca-trust-extracted\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317702 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317471 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6069cb-1eaa-41a3-818d-42c744eb74a9-image-registry-private-configuration\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.317702 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.317479 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6069cb-1eaa-41a3-818d-42c744eb74a9-registry-certificates\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:22:56.526609 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.526519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:56.629576 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.629536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:56.750713 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.750680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:56.800836 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.800798 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65"] Jul 07 02:22:56.801036 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.801023 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" Jul 07 02:22:56.801096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.801037 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" Jul 07 02:22:56.805429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.801469 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerName="registry" Jul 07 02:22:56.808723 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.808697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.810721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.810682 2564 generic.go:356] "Generic (PLEG): container finished" podID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" containerID="d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5" exitCode=0 Jul 07 02:22:56.810885 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.810808 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" Jul 07 02:22:56.810960 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.810805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" event={"ID":"ca6069cb-1eaa-41a3-818d-42c744eb74a9","Type":"ContainerDied","Data":"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5"} Jul 07 02:22:56.811015 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.810961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd78fc48-gdj65" event={"ID":"ca6069cb-1eaa-41a3-818d-42c744eb74a9","Type":"ContainerDied","Data":"fe02e1dff2803167a0388e3999bc4b0a489ff805496b31688cb1a925a7951bc8"} Jul 07 02:22:56.811062 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.811008 2564 scope.go:117] "RemoveContainer" containerID="d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5" Jul 07 02:22:56.811369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.811340 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h9qmc"] Jul 07 02:22:56.811545 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.811526 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h7swq\"" Jul 07 02:22:56.813056 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.813038 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 02:22:56.813713 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.813696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 02:22:56.813867 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.813762 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 02:22:56.815446 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.815423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.819031 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.819013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 02:22:56.819141 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.819069 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 02:22:56.819208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.819162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fpv7g\"" Jul 07 02:22:56.819410 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.819397 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 02:22:56.820414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.820396 2564 scope.go:117] "RemoveContainer" containerID="d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5" Jul 07 02:22:56.820741 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:56.820715 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5\": container with ID starting with d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5 not found: ID does not exist" containerID="d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5" Jul 07 02:22:56.820879 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.820757 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5"} err="failed to get container status \"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5\": rpc error: code = NotFound desc = could not find container \"d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5\": container with ID starting with d3f359d1543a33ededf8e5f6367a8a4f4172477f37ba86f7e34d2aa5371e7ee5 not found: ID does not exist" Jul 07 02:22:56.821138 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.821206 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.821255 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.821289 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwtl\" (UniqueName: \"kubernetes.io/projected/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-api-access-9qwtl\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.821327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.821370 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.821339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.823506 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.823486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65"] Jul 07 02:22:56.852378 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.852346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:56.865444 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.865413 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:56.871463 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.871435 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dd78fc48-gdj65"] Jul 07 02:22:56.922016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.921979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-metrics-client-ca\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-root\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-wtmp\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xjq\" (UniqueName: \"kubernetes.io/projected/30e2c724-5810-4d6e-bd70-3e1c519fd882-kube-api-access-h6xjq\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-sys\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwtl\" (UniqueName: \"kubernetes.io/projected/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-api-access-9qwtl\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-textfile\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:56.922716 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922716 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.922488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.922716 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:56.922647 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 02:22:56.922716 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:56.922712 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls podName:ccee1b2b-7739-40f1-98a8-7c5a11dc5833 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:57.422691392 +0000 UTC m=+302.180240416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-dvq65" (UID: "ccee1b2b-7739-40f1-98a8-7c5a11dc5833") : secret "kube-state-metrics-tls" not found Jul 07 02:22:56.923033 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.923013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.923345 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.923323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.924961 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.924939 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:56.931721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:56.931690 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwtl\" (UniqueName: \"kubernetes.io/projected/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-api-access-9qwtl\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:57.023929 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.023892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-wtmp\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.023929 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.023933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xjq\" (UniqueName: \"kubernetes.io/projected/30e2c724-5810-4d6e-bd70-3e1c519fd882-kube-api-access-h6xjq\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.023953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-sys\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.023977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-textfile\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024053 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-sys\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-wtmp\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:57.024145 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 02:22:57.024187 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024568 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:22:57.024218 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls podName:30e2c724-5810-4d6e-bd70-3e1c519fd882 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:57.524198705 +0000 UTC m=+302.281747708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls") pod "node-exporter-h9qmc" (UID: "30e2c724-5810-4d6e-bd70-3e1c519fd882") : secret "node-exporter-tls" not found Jul 07 02:22:57.024568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-metrics-client-ca\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-root\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024341 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e2c724-5810-4d6e-bd70-3e1c519fd882-root\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024481 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-textfile\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024751 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024660 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.024751 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.024712 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e2c724-5810-4d6e-bd70-3e1c519fd882-metrics-client-ca\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.026608 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.026588 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.042130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.042090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xjq\" (UniqueName: \"kubernetes.io/projected/30e2c724-5810-4d6e-bd70-3e1c519fd882-kube-api-access-h6xjq\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.113334 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.113241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.207137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.207102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.277515 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.277481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.400856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.400747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.428213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.428176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:57.430962 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.430930 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccee1b2b-7739-40f1-98a8-7c5a11dc5833-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dvq65\" (UID: \"ccee1b2b-7739-40f1-98a8-7c5a11dc5833\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:57.529263 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.529219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.531851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.531826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e2c724-5810-4d6e-bd70-3e1c519fd882-node-exporter-tls\") pod \"node-exporter-h9qmc\" (UID: \"30e2c724-5810-4d6e-bd70-3e1c519fd882\") " pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.623851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.623811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.724685 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.724588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" Jul 07 02:22:57.730209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.730176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h9qmc" Jul 07 02:22:57.743919 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:57.743881 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e2c724_5810_4d6e_bd70_3e1c519fd882.slice/crio-f47d01470fb0432224834e302089df16129b92efd74acfe2ea1cb90aa89bf273 WatchSource:0}: Error finding container f47d01470fb0432224834e302089df16129b92efd74acfe2ea1cb90aa89bf273: Status 404 returned error can't find the container with id f47d01470fb0432224834e302089df16129b92efd74acfe2ea1cb90aa89bf273 Jul 07 02:22:57.746215 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.746191 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:22:57.778904 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.778860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:57.792427 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.792388 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca6069cb-1eaa-41a3-818d-42c744eb74a9" path="/var/lib/kubelet/pods/ca6069cb-1eaa-41a3-818d-42c744eb74a9/volumes" Jul 07 02:22:57.815350 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.815311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9qmc" event={"ID":"30e2c724-5810-4d6e-bd70-3e1c519fd882","Type":"ContainerStarted","Data":"f47d01470fb0432224834e302089df16129b92efd74acfe2ea1cb90aa89bf273"} Jul 07 02:22:57.880304 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.880271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65"] Jul 07 02:22:57.883244 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:22:57.883212 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccee1b2b_7739_40f1_98a8_7c5a11dc5833.slice/crio-aaec61ec0963e8afb098bb5b99c6da02450f43b07046795401527491d53c65d2 WatchSource:0}: Error finding container aaec61ec0963e8afb098bb5b99c6da02450f43b07046795401527491d53c65d2: Status 404 returned error can't find the container with id aaec61ec0963e8afb098bb5b99c6da02450f43b07046795401527491d53c65d2 Jul 07 02:22:57.887142 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:57.887109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:58.009768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:58.009682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:58.822835 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:58.822770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" event={"ID":"ccee1b2b-7739-40f1-98a8-7c5a11dc5833","Type":"ContainerStarted","Data":"aaec61ec0963e8afb098bb5b99c6da02450f43b07046795401527491d53c65d2"} Jul 07 02:22:59.713660 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.713623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:59.762200 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.762159 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:59.762414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.762240 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:59.767215 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.767176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:59.799861 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.799822 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:59.829768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.829722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" event={"ID":"ccee1b2b-7739-40f1-98a8-7c5a11dc5833","Type":"ContainerStarted","Data":"3106b52b03ae27199beb8b07fce25cf2441de4ef0f4bd610d0426721433b2894"} Jul 07 02:22:59.830277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.829811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" event={"ID":"ccee1b2b-7739-40f1-98a8-7c5a11dc5833","Type":"ContainerStarted","Data":"47194954b72bd62869470e1286e6e6f640ca0ea85f0df12e7173e21cf1d3e82c"} Jul 07 02:22:59.830277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.829838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" event={"ID":"ccee1b2b-7739-40f1-98a8-7c5a11dc5833","Type":"ContainerStarted","Data":"921a7ea73aba22896b6264543a5744adbfc6b1444bbf54d6eb82c7857346b508"} Jul 07 02:22:59.832170 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.832134 2564 generic.go:356] "Generic (PLEG): container finished" podID="30e2c724-5810-4d6e-bd70-3e1c519fd882" containerID="a96891f5ee87f34dfa0bb331c0b9e4cff303ee3f48f866fba15710129167bbcf" exitCode=0 Jul 07 02:22:59.832338 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.832220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9qmc" event={"ID":"30e2c724-5810-4d6e-bd70-3e1c519fd882","Type":"ContainerDied","Data":"a96891f5ee87f34dfa0bb331c0b9e4cff303ee3f48f866fba15710129167bbcf"} Jul 07 02:22:59.841133 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.841092 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:22:59.856241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.856183 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dvq65" podStartSLOduration=2.434191507 podStartE2EDuration="3.856161879s" totalImagesPullingTime="1.421970372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:59.855384613 +0000 UTC m=+304.612933668" watchObservedRunningTime="2026-07-07 02:22:59.856161879 +0000 UTC m=+304.613710904" Jul 07 02:22:59.867510 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.867469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:22:59.911083 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:22:59.911045 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:23:00.025844 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.025810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:00.125354 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.125320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:00.225177 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.225087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:00.327520 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.327480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:00.838429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.838384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9qmc" event={"ID":"30e2c724-5810-4d6e-bd70-3e1c519fd882","Type":"ContainerStarted","Data":"070b0bc65c5c3f8be3388558abbb1c3ee519ca282814874cb41603c8d66eff0f"} Jul 07 02:23:00.838925 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.838447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9qmc" event={"ID":"30e2c724-5810-4d6e-bd70-3e1c519fd882","Type":"ContainerStarted","Data":"d4e1b33340ecea6c2f88878f251cfe81b3b6e2b74eb554c8705a24beee91bac1"} Jul 07 02:23:00.861892 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:00.861838 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h9qmc" podStartSLOduration=3.750644262 podStartE2EDuration="4.861820041s" totalImagesPullingTime="1.02235234s" totalInitContainerRuntime="88.823439ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:00.859819836 +0000 UTC m=+305.617368875" watchObservedRunningTime="2026-07-07 02:23:00.861820041 +0000 UTC m=+305.619369065" Jul 07 02:23:01.547077 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.547038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:01.554764 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.553392 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9"] Jul 07 02:23:01.557186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.557163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.561442 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.561400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 02:23:01.561442 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.561415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h9v6b\"" Jul 07 02:23:01.569739 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.569703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9"] Jul 07 02:23:01.638326 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.638279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:01.664810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.664753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8ff000-61ee-43ae-917e-1b7511223832-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.664993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.664846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8w6\" (UniqueName: \"kubernetes.io/projected/8c8ff000-61ee-43ae-917e-1b7511223832-kube-api-access-cv8w6\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.710944 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.710906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:01.765414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.765372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8ff000-61ee-43ae-917e-1b7511223832-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.765622 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.765428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8w6\" (UniqueName: \"kubernetes.io/projected/8c8ff000-61ee-43ae-917e-1b7511223832-kube-api-access-cv8w6\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.768370 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.768338 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8ff000-61ee-43ae-917e-1b7511223832-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.776709 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.776673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8w6\" (UniqueName: \"kubernetes.io/projected/8c8ff000-61ee-43ae-917e-1b7511223832-kube-api-access-cv8w6\") pod \"monitoring-plugin-d5dcff7cf-k29c9\" (UID: \"8c8ff000-61ee-43ae-917e-1b7511223832\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:01.868156 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:01.868075 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:02.010441 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:02.010403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9"] Jul 07 02:23:02.011688 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:02.011659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:02.013973 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:23:02.013937 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8ff000_61ee_43ae_917e_1b7511223832.slice/crio-7b64896f707f45c8184e2238356fea9a7dee67dd9adb9b177b4f39244154bef7 WatchSource:0}: Error finding container 7b64896f707f45c8184e2238356fea9a7dee67dd9adb9b177b4f39244154bef7: Status 404 returned error can't find the container with id 7b64896f707f45c8184e2238356fea9a7dee67dd9adb9b177b4f39244154bef7 Jul 07 02:23:02.109414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:02.109374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:02.849488 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:02.849447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" event={"ID":"8c8ff000-61ee-43ae-917e-1b7511223832","Type":"ContainerStarted","Data":"7b64896f707f45c8184e2238356fea9a7dee67dd9adb9b177b4f39244154bef7"} Jul 07 02:23:03.036823 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:03.036734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:03.109742 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:03.109539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:04.859978 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:04.859937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" event={"ID":"8c8ff000-61ee-43ae-917e-1b7511223832","Type":"ContainerStarted","Data":"563ee51e19ae9421174fe13013e9bf6557924727210d3428fe82c0f3df94c2c5"} Jul 07 02:23:04.860433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:04.860198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:04.865102 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:04.865071 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" Jul 07 02:23:04.877669 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:04.877575 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k29c9" podStartSLOduration=1.911156916 podStartE2EDuration="3.877554386s" totalImagesPullingTime="1.96639747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:04.876357601 +0000 UTC m=+309.633906626" watchObservedRunningTime="2026-07-07 02:23:04.877554386 +0000 UTC m=+309.635103411" Jul 07 02:23:06.957689 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:06.957642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:07.018933 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:07.018897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:08.172604 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:08.172560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:08.273159 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:08.273109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:08.785574 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:08.785536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:23:11.451050 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:11.450998 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" podUID="a25cfe3c-c992-46d5-a7e0-a24780df4e54" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 02:23:13.173513 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:13.173468 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:23:13.311403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:13.311366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:13.415292 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:13.415256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:13.514214 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:13.514113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:13.618720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:13.618670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:16.052129 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:16.052095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:16.151161 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:16.151117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:21.450509 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:21.450468 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" podUID="a25cfe3c-c992-46d5-a7e0-a24780df4e54" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 02:23:24.938185 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:24.938143 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8564dc94c8-psh89" podUID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" containerName="console" containerID="cri-o://9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c" gracePeriod=15 Jul 07 02:23:25.196573 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.196497 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:23:25.279965 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.279910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.279991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280060 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280090 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280138 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnkg6\" (UniqueName: \"kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6\") pod \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\" (UID: \"c5edbcfb-6d24-409b-b183-a8e4d92af5a2\") " Jul 07 02:23:25.280424 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280396 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config" (OuterVolumeSpecName: "console-config") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:25.280491 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280469 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:25.280491 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.280466 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:25.282821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.282755 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:23:25.282821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.282769 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:23:25.282821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.282756 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6" (OuterVolumeSpecName: "kube-api-access-bnkg6") pod "c5edbcfb-6d24-409b-b183-a8e4d92af5a2" (UID: "c5edbcfb-6d24-409b-b183-a8e4d92af5a2"). InnerVolumeSpecName "kube-api-access-bnkg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:23:25.381884 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381831 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkg6\" (UniqueName: \"kubernetes.io/projected/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-kube-api-access-bnkg6\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.381884 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381868 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.381884 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381878 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-oauth-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.381884 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381886 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-oauth-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.381884 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381895 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-service-ca\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.382186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.381903 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5edbcfb-6d24-409b-b183-a8e4d92af5a2-console-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:25.938421 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.938381 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" containerID="9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c" exitCode=0 Jul 07 02:23:25.938919 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.938445 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8564dc94c8-psh89" Jul 07 02:23:25.938919 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.938472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564dc94c8-psh89" event={"ID":"c5edbcfb-6d24-409b-b183-a8e4d92af5a2","Type":"ContainerDied","Data":"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c"} Jul 07 02:23:25.938919 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.938534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8564dc94c8-psh89" event={"ID":"c5edbcfb-6d24-409b-b183-a8e4d92af5a2","Type":"ContainerDied","Data":"ef34bee0d70d1a7a41d456c78e32fe9a2ae5efcba40daa9c087a029799d6b15a"} Jul 07 02:23:25.938919 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.938564 2564 scope.go:117] "RemoveContainer" containerID="9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c" Jul 07 02:23:25.947473 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.947453 2564 scope.go:117] "RemoveContainer" containerID="9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c" Jul 07 02:23:25.947815 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:23:25.947777 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c\": container with ID starting with 9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c not found: ID does not exist" containerID="9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c" Jul 07 02:23:25.947883 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.947827 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c"} err="failed to get container status \"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c\": rpc error: code = NotFound desc = could not find container \"9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c\": container with ID starting with 9155a0cdfe9f8315bb9146f3b8dd3c4d298de186f8dd07cfdafbfe50f813d53c not found: ID does not exist" Jul 07 02:23:25.955457 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.955421 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:23:25.959807 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:25.959749 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8564dc94c8-psh89"] Jul 07 02:23:26.502896 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:26.502865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:26.600741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:26.600702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:27.790464 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:27.790416 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" path="/var/lib/kubelet/pods/c5edbcfb-6d24-409b-b183-a8e4d92af5a2/volumes" Jul 07 02:23:31.451108 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.451069 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" podUID="a25cfe3c-c992-46d5-a7e0-a24780df4e54" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 02:23:31.451508 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.451151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" Jul 07 02:23:31.451639 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.451619 2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"5c324caff6f5d85738f661c557f2382f933767e46b38ccf7ca2bfe42a2d37d7a"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 07 02:23:31.451676 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.451659 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" podUID="a25cfe3c-c992-46d5-a7e0-a24780df4e54" containerName="service-proxy" containerID="cri-o://5c324caff6f5d85738f661c557f2382f933767e46b38ccf7ca2bfe42a2d37d7a" gracePeriod=30 Jul 07 02:23:31.965336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.965303 2564 generic.go:356] "Generic (PLEG): container finished" podID="a25cfe3c-c992-46d5-a7e0-a24780df4e54" containerID="5c324caff6f5d85738f661c557f2382f933767e46b38ccf7ca2bfe42a2d37d7a" exitCode=2 Jul 07 02:23:31.965571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.965369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerDied","Data":"5c324caff6f5d85738f661c557f2382f933767e46b38ccf7ca2bfe42a2d37d7a"} Jul 07 02:23:31.965571 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:31.965407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7979bd9-flcld" event={"ID":"a25cfe3c-c992-46d5-a7e0-a24780df4e54","Type":"ContainerStarted","Data":"b44951b08b849fa0d6a4e2582e8041d83e220138224cfb4f1aa94e4c3c53c29f"} Jul 07 02:23:38.195745 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.195686 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84c6494674-sp8mh" podUID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" containerName="console" containerID="cri-o://ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1" gracePeriod=15 Jul 07 02:23:38.438178 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.438151 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:23:38.601926 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.601886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.601926 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.601929 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.601970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.601993 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602025 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602059 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpdr8\" (UniqueName: \"kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602092 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert\") pod \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\" (UID: \"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a\") " Jul 07 02:23:38.602431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602403 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca" (OuterVolumeSpecName: "service-ca") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:38.602502 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602468 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config" (OuterVolumeSpecName: "console-config") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:38.602502 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602482 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:38.602502 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602481 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:23:38.602677 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602660 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-service-ca\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.602711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602686 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-trusted-ca-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.602711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602701 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-oauth-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.602771 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.602715 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.604371 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.604347 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:23:38.604895 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.604860 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:23:38.604976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.604865 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8" (OuterVolumeSpecName: "kube-api-access-vpdr8") pod "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" (UID: "6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a"). InnerVolumeSpecName "kube-api-access-vpdr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:23:38.703569 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.703519 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-oauth-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.703569 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.703563 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdr8\" (UniqueName: \"kubernetes.io/projected/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-kube-api-access-vpdr8\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.703569 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.703576 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a-console-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:23:38.996449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.996344 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" containerID="ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1" exitCode=0 Jul 07 02:23:38.996449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.996392 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6494674-sp8mh" Jul 07 02:23:38.996449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.996403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6494674-sp8mh" event={"ID":"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a","Type":"ContainerDied","Data":"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1"} Jul 07 02:23:38.996724 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.996473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6494674-sp8mh" event={"ID":"6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a","Type":"ContainerDied","Data":"49bd1b096426ae96ee830927b5c0858d3582f10392d04c9ea93b8a67c2ee8fcc"} Jul 07 02:23:38.996724 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:38.996501 2564 scope.go:117] "RemoveContainer" containerID="ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1" Jul 07 02:23:39.006240 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:39.006220 2564 scope.go:117] "RemoveContainer" containerID="ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1" Jul 07 02:23:39.006596 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:23:39.006577 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1\": container with ID starting with ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1 not found: ID does not exist" containerID="ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1" Jul 07 02:23:39.006673 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:39.006610 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1"} err="failed to get container status \"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1\": rpc error: code = NotFound desc = could not find container \"ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1\": container with ID starting with ae7837d1be2fbc62dc40fa03d4ae012317d1bbdb7b9ebdcebb333074d052c9a1 not found: ID does not exist" Jul 07 02:23:39.018249 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:39.018209 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:23:39.024833 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:39.024799 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c6494674-sp8mh"] Jul 07 02:23:39.790258 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:39.790223 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" path="/var/lib/kubelet/pods/6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a/volumes" Jul 07 02:23:41.503480 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:41.503441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:23:41.570969 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:23:41.570938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:00.431678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:00.431638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:00.503028 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:00.502985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:09.649184 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:09.649141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:24:09.651883 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:09.651852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b9e554-ea7a-4647-b208-0cbb59badeb6-metrics-certs\") pod \"network-metrics-daemon-t4vsh\" (UID: \"b7b9e554-ea7a-4647-b208-0cbb59badeb6\") " pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:24:09.689799 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:09.689744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\"" Jul 07 02:24:09.697497 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:09.697464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vsh" Jul 07 02:24:09.846838 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:09.846777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4vsh"] Jul 07 02:24:09.849676 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:24:09.849641 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b9e554_ea7a_4647_b208_0cbb59badeb6.slice/crio-837134fd9fb391c9a552a105e7d07d85f8a5d49cfde06ca79de923c5c0dbd2bd WatchSource:0}: Error finding container 837134fd9fb391c9a552a105e7d07d85f8a5d49cfde06ca79de923c5c0dbd2bd: Status 404 returned error can't find the container with id 837134fd9fb391c9a552a105e7d07d85f8a5d49cfde06ca79de923c5c0dbd2bd Jul 07 02:24:10.102830 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:10.102766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vsh" event={"ID":"b7b9e554-ea7a-4647-b208-0cbb59badeb6","Type":"ContainerStarted","Data":"837134fd9fb391c9a552a105e7d07d85f8a5d49cfde06ca79de923c5c0dbd2bd"} Jul 07 02:24:11.108798 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:11.108744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vsh" event={"ID":"b7b9e554-ea7a-4647-b208-0cbb59badeb6","Type":"ContainerStarted","Data":"7c5683f704618eb6032964d57a475fac09698d0703feafc873534ae4a7f8a13e"} Jul 07 02:24:12.114761 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:12.114718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vsh" event={"ID":"b7b9e554-ea7a-4647-b208-0cbb59badeb6","Type":"ContainerStarted","Data":"e3cbfccb3a648254a0cbd2f81911a04b72c962416e4ba4be3f35ec170c8856e8"} Jul 07 02:24:12.134048 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:12.133987 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t4vsh" podStartSLOduration=376.03902142 podStartE2EDuration="6m17.133970026s" totalImagesPullingTime="1.094948606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:12.132108608 +0000 UTC m=+376.889657649" watchObservedRunningTime="2026-07-07 02:24:12.133970026 +0000 UTC m=+376.891519051" Jul 07 02:24:12.228077 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:12.228028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:12.292678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:12.292638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:16.040114 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:16.040071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:16.246646 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:16.246608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:24.337893 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:24.337852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:24.441237 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:24.441202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:25.293044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:25.293001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.156094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156054 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156301 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" containerName="console" Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156313 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" containerName="console" Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156327 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" containerName="console" Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156333 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" containerName="console" Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156378 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5edbcfb-6d24-409b-b183-a8e4d92af5a2" containerName="console" Jul 07 02:24:27.156703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.156387 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6f6aee-cd9d-4d4f-a4bc-5bb85ec9261a" containerName="console" Jul 07 02:24:27.158244 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.158215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.161039 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.161009 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 02:24:27.162496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162340 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 02:24:27.162496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162424 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 02:24:27.162496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162427 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 02:24:27.162754 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162508 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bvcr9\"" Jul 07 02:24:27.162754 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162508 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 02:24:27.162754 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 02:24:27.162754 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.162752 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 02:24:27.167809 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.167759 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 02:24:27.171703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.171670 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:24:27.207902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.207862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.283721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rh4\" (UniqueName: \"kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.283964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.283860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.307254 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.307218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.384365 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rh4\" (UniqueName: \"kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.384580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.384565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.385343 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.385311 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.385343 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.385342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.385570 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.385547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.385951 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.385931 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.387238 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.387208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.387329 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.387282 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.393564 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.393530 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rh4\" (UniqueName: \"kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4\") pod \"console-ff6d6597f-rg84t\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.408288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.408194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.466159 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.466109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:27.472975 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.472939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:27.656210 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.656176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:24:27.660475 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:24:27.660364 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73ecfd1_2b67_4b56_81c6_a5027d9c2fd9.slice/crio-8cc4d441eb0c79f6199cf413cff692fa49ed59b3dd0ab6795e0bcd86bc6b5848 WatchSource:0}: Error finding container 8cc4d441eb0c79f6199cf413cff692fa49ed59b3dd0ab6795e0bcd86bc6b5848: Status 404 returned error can't find the container with id 8cc4d441eb0c79f6199cf413cff692fa49ed59b3dd0ab6795e0bcd86bc6b5848 Jul 07 02:24:27.953366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:27.953275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:28.011188 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:28.011149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:28.174922 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:28.174888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff6d6597f-rg84t" event={"ID":"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9","Type":"ContainerStarted","Data":"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8"} Jul 07 02:24:28.175320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:28.174939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff6d6597f-rg84t" event={"ID":"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9","Type":"ContainerStarted","Data":"8cc4d441eb0c79f6199cf413cff692fa49ed59b3dd0ab6795e0bcd86bc6b5848"} Jul 07 02:24:28.194669 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:28.194614 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff6d6597f-rg84t" podStartSLOduration=1.194597217 podStartE2EDuration="1.194597217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:28.193843201 +0000 UTC m=+392.951392238" watchObservedRunningTime="2026-07-07 02:24:28.194597217 +0000 UTC m=+392.952146242" Jul 07 02:24:37.474087 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:37.474046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:37.474540 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:37.474111 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:37.479142 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:37.479112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:38.217320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:38.217287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:24:38.372173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:38.372131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:38.474849 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:38.474739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:38.531214 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:38.531175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:24:51.008563 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:24:51.008522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:29.070067 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:29.069984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:31.260091 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:31.260049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:31.320771 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:31.320731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:43.150940 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:43.150903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:43.207866 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:43.207817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:47.353535 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.353493 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:25:47.356525 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.356500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.368657 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.368624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:25:47.456431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.456395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:47.492964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.492924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.492964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.492970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.493186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.493001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.493186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.493062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.493186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.493087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.493186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.493131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcr2\" (UniqueName: \"kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.493325 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.493198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.594537 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcr2\" (UniqueName: \"kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.594537 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.594731 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.594768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.594951 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595006 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.594983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595049 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.595010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595467 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.595433 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595467 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.595462 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.595496 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.595628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.595522 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.597812 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.597761 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.597903 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.597818 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.604157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.604087 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcr2\" (UniqueName: \"kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2\") pod \"console-7678dd94b4-gbcrn\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.666333 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.666290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:47.815614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:47.815588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:25:47.818396 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:25:47.818366 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281eb1a8_fe8b_4064_a308_9ffbc497951a.slice/crio-6eeafd8fefc9f4932dd8cb4ef798fbdac104503016e522b78802f09831db14b4 WatchSource:0}: Error finding container 6eeafd8fefc9f4932dd8cb4ef798fbdac104503016e522b78802f09831db14b4: Status 404 returned error can't find the container with id 6eeafd8fefc9f4932dd8cb4ef798fbdac104503016e522b78802f09831db14b4 Jul 07 02:25:48.467403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:48.467365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7678dd94b4-gbcrn" event={"ID":"281eb1a8-fe8b-4064-a308-9ffbc497951a","Type":"ContainerStarted","Data":"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc"} Jul 07 02:25:48.467909 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:48.467417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7678dd94b4-gbcrn" event={"ID":"281eb1a8-fe8b-4064-a308-9ffbc497951a","Type":"ContainerStarted","Data":"6eeafd8fefc9f4932dd8cb4ef798fbdac104503016e522b78802f09831db14b4"} Jul 07 02:25:48.487366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:48.487307 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7678dd94b4-gbcrn" podStartSLOduration=1.487287861 podStartE2EDuration="1.487287861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:48.486395003 +0000 UTC m=+473.243944048" watchObservedRunningTime="2026-07-07 02:25:48.487287861 +0000 UTC m=+473.244836886" Jul 07 02:25:56.502313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:56.502275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:56.601031 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:56.600983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:56.660491 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:56.660455 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:56.900720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:56.900678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:57.666438 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:57.666391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:57.666892 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:57.666456 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:57.671460 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:57.671428 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:58.507146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:58.507114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:25:58.552085 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:58.552034 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:25:58.665708 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:58.665676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:58.766577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:58.766488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:25:58.821994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:25:58.821955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:23.574234 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.574185 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-ff6d6597f-rg84t" podUID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" containerName="console" containerID="cri-o://60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8" gracePeriod=15 Jul 07 02:26:23.826305 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.826228 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:26:23.854277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.854277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.854550 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854310 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.854550 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854340 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.854731 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854707 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca" (OuterVolumeSpecName: "service-ca") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:26:23.854907 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.854879 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:26:23.856835 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.856777 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:26:23.856835 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.856810 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:26:23.954897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.954853 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5rh4\" (UniqueName: \"kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.954897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.954900 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.955140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.954988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert\") pod \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\" (UID: \"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9\") " Jul 07 02:26:23.955140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955102 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-trusted-ca-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:23.955140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955113 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-oauth-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:23.955140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955128 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:23.955140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955140 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-service-ca\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:23.955400 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955267 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config" (OuterVolumeSpecName: "console-config") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:26:23.955400 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.955376 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:26:23.957225 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:23.957189 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4" (OuterVolumeSpecName: "kube-api-access-l5rh4") pod "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" (UID: "f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9"). InnerVolumeSpecName "kube-api-access-l5rh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:26:24.055662 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.055622 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-oauth-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:24.055662 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.055662 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rh4\" (UniqueName: \"kubernetes.io/projected/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-kube-api-access-l5rh4\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:24.055928 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.055675 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9-console-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:24.598218 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.598175 2564 generic.go:356] "Generic (PLEG): container finished" podID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" containerID="60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8" exitCode=0 Jul 07 02:26:24.598632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.598226 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff6d6597f-rg84t" Jul 07 02:26:24.598632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.598261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff6d6597f-rg84t" event={"ID":"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9","Type":"ContainerDied","Data":"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8"} Jul 07 02:26:24.598632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.598322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff6d6597f-rg84t" event={"ID":"f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9","Type":"ContainerDied","Data":"8cc4d441eb0c79f6199cf413cff692fa49ed59b3dd0ab6795e0bcd86bc6b5848"} Jul 07 02:26:24.598632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.598344 2564 scope.go:117] "RemoveContainer" containerID="60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8" Jul 07 02:26:24.607365 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.607349 2564 scope.go:117] "RemoveContainer" containerID="60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8" Jul 07 02:26:24.607691 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:26:24.607673 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8\": container with ID starting with 60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8 not found: ID does not exist" containerID="60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8" Jul 07 02:26:24.607738 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.607700 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8"} err="failed to get container status \"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8\": rpc error: code = NotFound desc = could not find container \"60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8\": container with ID starting with 60c3471ddbd6e41258522f2dd705d5950eb1db35531d68b2b465bd406eb95df8 not found: ID does not exist" Jul 07 02:26:24.621197 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.621153 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:26:24.625044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:24.625011 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff6d6597f-rg84t"] Jul 07 02:26:25.790584 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:25.790540 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" path="/var/lib/kubelet/pods/f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9/volumes" Jul 07 02:26:31.963258 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:31.963217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:32.717238 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:32.717205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:32.811910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:32.811874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:33.030112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:33.030000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:33.129016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:33.128978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:33.228665 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:33.228631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:26:34.066854 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.066813 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp"] Jul 07 02:26:34.067350 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.067175 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" containerName="console" Jul 07 02:26:34.067350 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.067191 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" containerName="console" Jul 07 02:26:34.067350 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.067262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73ecfd1-2b67-4b56-81c6-a5027d9c2fd9" containerName="console" Jul 07 02:26:34.071977 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.071946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.074954 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.074924 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 02:26:34.075994 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.075975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rqcdc\"" Jul 07 02:26:34.076092 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.076021 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 02:26:34.087945 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.087913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp"] Jul 07 02:26:34.128805 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.128742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbf7\" (UniqueName: \"kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.128805 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.128806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.129070 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.128896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.230288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.230233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbf7\" (UniqueName: \"kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.230288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.230305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.230554 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.230337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.230843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.230825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.230926 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.230857 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.239629 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.239596 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbf7\" (UniqueName: \"kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.381950 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.381848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:34.525980 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.525948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp"] Jul 07 02:26:34.529028 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:26:34.528990 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c25bb9_0ec7_43b2_9c91_fe9aceb3b8be.slice/crio-9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9 WatchSource:0}: Error finding container 9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9: Status 404 returned error can't find the container with id 9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9 Jul 07 02:26:34.638189 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:34.638099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" event={"ID":"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be","Type":"ContainerStarted","Data":"9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9"} Jul 07 02:26:41.668036 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:41.667941 2564 generic.go:356] "Generic (PLEG): container finished" podID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerID="aebb559cabb06f59df701a0c08e6de7fd183a7e9279104f1bef871203f7900e6" exitCode=0 Jul 07 02:26:41.668416 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:41.668037 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" event={"ID":"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be","Type":"ContainerDied","Data":"aebb559cabb06f59df701a0c08e6de7fd183a7e9279104f1bef871203f7900e6"} Jul 07 02:26:45.685209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:45.685175 2564 generic.go:356] "Generic (PLEG): container finished" podID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerID="f5ad1804d2c3594957733377fbcbea8b3ea968d4b385601fdba24c8d8ed85fd4" exitCode=0 Jul 07 02:26:45.685651 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:45.685222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" event={"ID":"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be","Type":"ContainerDied","Data":"f5ad1804d2c3594957733377fbcbea8b3ea968d4b385601fdba24c8d8ed85fd4"} Jul 07 02:26:53.721704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:53.721666 2564 generic.go:356] "Generic (PLEG): container finished" podID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerID="43e8d46da8657ce6876bf764e4d644331df851c85beb45bca298af72fe05688d" exitCode=0 Jul 07 02:26:53.722146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:53.721749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" event={"ID":"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be","Type":"ContainerDied","Data":"43e8d46da8657ce6876bf764e4d644331df851c85beb45bca298af72fe05688d"} Jul 07 02:26:54.857677 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.857641 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:54.903218 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.903177 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjbf7\" (UniqueName: \"kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7\") pod \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " Jul 07 02:26:54.903386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.903260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util\") pod \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " Jul 07 02:26:54.903386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.903299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle\") pod \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\" (UID: \"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be\") " Jul 07 02:26:54.903982 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.903950 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle" (OuterVolumeSpecName: "bundle") pod "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" (UID: "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:26:54.905624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.905588 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7" (OuterVolumeSpecName: "kube-api-access-zjbf7") pod "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" (UID: "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be"). InnerVolumeSpecName "kube-api-access-zjbf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:26:54.907383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:54.907355 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util" (OuterVolumeSpecName: "util") pod "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" (UID: "d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:26:55.004586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.004493 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:55.004586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.004526 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbf7\" (UniqueName: \"kubernetes.io/projected/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-kube-api-access-zjbf7\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:55.004586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.004536 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:26:55.731851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.731825 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" Jul 07 02:26:55.732029 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.731829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589vfjrp" event={"ID":"d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be","Type":"ContainerDied","Data":"9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9"} Jul 07 02:26:55.732029 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:26:55.731938 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f36404c10a3b8ce3fdbb7237fbd1a4ecba0ea83254ee7a9291cf58f419f06f9" Jul 07 02:27:00.495128 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:00.495086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:00.592057 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:00.592006 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:02.115278 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115246 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk"] Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115483 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="util" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115493 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="util" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115501 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="extract" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115506 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="extract" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115522 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="pull" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115527 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="pull" Jul 07 02:27:02.115663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.115574 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c25bb9-0ec7-43b2-9c91-fe9aceb3b8be" containerName="extract" Jul 07 02:27:02.118574 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.118554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.122273 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.122252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 07 02:27:02.122580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.122558 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2q2dp\"" Jul 07 02:27:02.122769 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.122754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 07 02:27:02.124462 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.124444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 07 02:27:02.140595 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.140551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk"] Jul 07 02:27:02.260611 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.260573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e370b31-62ea-43eb-8adc-a99bf5a3bade-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.260611 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.260616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxts\" (UniqueName: \"kubernetes.io/projected/5e370b31-62ea-43eb-8adc-a99bf5a3bade-kube-api-access-hlxts\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.362050 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.362010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e370b31-62ea-43eb-8adc-a99bf5a3bade-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.362050 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.362053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxts\" (UniqueName: \"kubernetes.io/projected/5e370b31-62ea-43eb-8adc-a99bf5a3bade-kube-api-access-hlxts\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.364572 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.364546 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e370b31-62ea-43eb-8adc-a99bf5a3bade-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.380625 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.380538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxts\" (UniqueName: \"kubernetes.io/projected/5e370b31-62ea-43eb-8adc-a99bf5a3bade-kube-api-access-hlxts\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk\" (UID: \"5e370b31-62ea-43eb-8adc-a99bf5a3bade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.428388 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.428339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:02.598229 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.598198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk"] Jul 07 02:27:02.601715 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:27:02.601684 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e370b31_62ea_43eb_8adc_a99bf5a3bade.slice/crio-4036130a279bf74688b5b809cfc1de9d477c52c24c9e71404160f5869df3c9d8 WatchSource:0}: Error finding container 4036130a279bf74688b5b809cfc1de9d477c52c24c9e71404160f5869df3c9d8: Status 404 returned error can't find the container with id 4036130a279bf74688b5b809cfc1de9d477c52c24c9e71404160f5869df3c9d8 Jul 07 02:27:02.757375 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:02.757293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" event={"ID":"5e370b31-62ea-43eb-8adc-a99bf5a3bade","Type":"ContainerStarted","Data":"4036130a279bf74688b5b809cfc1de9d477c52c24c9e71404160f5869df3c9d8"} Jul 07 02:27:07.790578 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:07.790528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" event={"ID":"5e370b31-62ea-43eb-8adc-a99bf5a3bade","Type":"ContainerStarted","Data":"5be9371f0d8b3f0ea3efafc305206afa44e85b7f19b5b7e008311625d59e593c"} Jul 07 02:27:07.791117 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:07.790596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:07.817453 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:07.817390 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" podStartSLOduration=0.799145521 podStartE2EDuration="5.817368793s" totalImagesPullingTime="5.018223272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:07.814985628 +0000 UTC m=+552.572534672" watchObservedRunningTime="2026-07-07 02:27:07.817368793 +0000 UTC m=+552.574917817" Jul 07 02:27:08.119444 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.119402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:08.538800 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.538744 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-gd78l"] Jul 07 02:27:08.541146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.541116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.544155 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.544124 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 07 02:27:08.545519 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.545495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vwjlh\"" Jul 07 02:27:08.545710 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.545691 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 07 02:27:08.556144 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.556113 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-gd78l"] Jul 07 02:27:08.559345 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.559308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:08.613715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.613666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tclc\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-kube-api-access-2tclc\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.613715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.613713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-cabundle0\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.613984 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.613740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.710192 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.710154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:08.714649 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.714612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tclc\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-kube-api-access-2tclc\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.714876 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.714661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-cabundle0\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.714876 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.714703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.714876 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:08.714841 2564 secret.go:281] references non-existent secret key: ca.crt Jul 07 02:27:08.714876 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:08.714858 2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 07 02:27:08.714876 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:08.714868 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-gd78l: references non-existent secret key: ca.crt Jul 07 02:27:08.715131 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:08.714925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates podName:47194965-f6c3-4a51-a3b4-dc8883f6eb5e nodeName:}" failed. No retries permitted until 2026-07-07 02:27:09.214906283 +0000 UTC m=+553.972455286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates") pod "keda-operator-68bff59c-gd78l" (UID: "47194965-f6c3-4a51-a3b4-dc8883f6eb5e") : references non-existent secret key: ca.crt Jul 07 02:27:08.715539 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.715513 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-cabundle0\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.734666 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.734621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tclc\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-kube-api-access-2tclc\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:08.893768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:08.893658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:09.039983 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.039940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:09.219013 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.218917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:09.219171 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:09.219093 2564 secret.go:281] references non-existent secret key: ca.crt Jul 07 02:27:09.219171 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:09.219117 2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 07 02:27:09.219171 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:09.219127 2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-gd78l: references non-existent secret key: ca.crt Jul 07 02:27:09.219273 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:27:09.219201 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates podName:47194965-f6c3-4a51-a3b4-dc8883f6eb5e nodeName:}" failed. No retries permitted until 2026-07-07 02:27:10.219183607 +0000 UTC m=+554.976732609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates") pod "keda-operator-68bff59c-gd78l" (UID: "47194965-f6c3-4a51-a3b4-dc8883f6eb5e") : references non-existent secret key: ca.crt Jul 07 02:27:09.737193 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.737146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:09.751732 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.751702 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-4dsts"] Jul 07 02:27:09.754484 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.754451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.758648 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.758622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 07 02:27:09.775028 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.774982 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-4dsts"] Jul 07 02:27:09.823225 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.823179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-certificates\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.823441 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.823268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxz5c\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-kube-api-access-cxz5c\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.882257 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.882215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:09.923964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.923915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxz5c\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-kube-api-access-cxz5c\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.924393 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.924091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-certificates\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.927269 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.927235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-certificates\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:09.938474 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:09.938438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxz5c\" (UniqueName: \"kubernetes.io/projected/34e1e460-cb6a-4bc0-af0a-d91295cbb55b-kube-api-access-cxz5c\") pod \"keda-admission-5fbd5c4644-4dsts\" (UID: \"34e1e460-cb6a-4bc0-af0a-d91295cbb55b\") " pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:10.030935 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.030825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:10.066953 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.066900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:10.227234 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.227194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:10.230308 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.230266 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/47194965-f6c3-4a51-a3b4-dc8883f6eb5e-certificates\") pod \"keda-operator-68bff59c-gd78l\" (UID: \"47194965-f6c3-4a51-a3b4-dc8883f6eb5e\") " pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:10.251322 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.251285 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-4dsts"] Jul 07 02:27:10.254742 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:27:10.254706 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e1e460_cb6a_4bc0_af0a_d91295cbb55b.slice/crio-091f6ad637899a73443614a93c9e7c74dcaf1c9a9f20c97d3c7987e2f27c6416 WatchSource:0}: Error finding container 091f6ad637899a73443614a93c9e7c74dcaf1c9a9f20c97d3c7987e2f27c6416: Status 404 returned error can't find the container with id 091f6ad637899a73443614a93c9e7c74dcaf1c9a9f20c97d3c7987e2f27c6416 Jul 07 02:27:10.352747 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.352707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:10.354364 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.354333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:10.527946 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.527909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-gd78l"] Jul 07 02:27:10.531973 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:27:10.531934 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47194965_f6c3_4a51_a3b4_dc8883f6eb5e.slice/crio-b97e69b4cf9c1eafcd6fd2712e0ba460747d7874ea09144fe26926551acb2381 WatchSource:0}: Error finding container b97e69b4cf9c1eafcd6fd2712e0ba460747d7874ea09144fe26926551acb2381: Status 404 returned error can't find the container with id b97e69b4cf9c1eafcd6fd2712e0ba460747d7874ea09144fe26926551acb2381 Jul 07 02:27:10.616945 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.616853 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:10.777547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.777506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:10.799972 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.799928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-gd78l" event={"ID":"47194965-f6c3-4a51-a3b4-dc8883f6eb5e","Type":"ContainerStarted","Data":"b97e69b4cf9c1eafcd6fd2712e0ba460747d7874ea09144fe26926551acb2381"} Jul 07 02:27:10.801210 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.801179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" event={"ID":"34e1e460-cb6a-4bc0-af0a-d91295cbb55b","Type":"ContainerStarted","Data":"091f6ad637899a73443614a93c9e7c74dcaf1c9a9f20c97d3c7987e2f27c6416"} Jul 07 02:27:10.872179 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:10.872090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:12.811170 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:12.811136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" event={"ID":"34e1e460-cb6a-4bc0-af0a-d91295cbb55b","Type":"ContainerStarted","Data":"8340e81ba418fea128c4ed37529bf24e11cc6f29a0d406b95c3a9fed41b4a9a5"} Jul 07 02:27:12.811578 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:12.811278 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:12.829073 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:12.828847 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" podStartSLOduration=2.053467685 podStartE2EDuration="3.828822745s" totalImagesPullingTime="1.77535506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:12.828643979 +0000 UTC m=+557.586193031" watchObservedRunningTime="2026-07-07 02:27:12.828822745 +0000 UTC m=+557.586371765" Jul 07 02:27:14.819993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:14.819956 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-gd78l" event={"ID":"47194965-f6c3-4a51-a3b4-dc8883f6eb5e","Type":"ContainerStarted","Data":"41adfd5b65e32007bb8db69722534b57f6b53c17631d88554637557379e9f2fb"} Jul 07 02:27:14.820430 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:14.820077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:14.846586 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:14.846521 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-gd78l" podStartSLOduration=2.918521236 podStartE2EDuration="6.846498735s" totalImagesPullingTime="3.927977499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:14.844749307 +0000 UTC m=+559.602298341" watchObservedRunningTime="2026-07-07 02:27:14.846498735 +0000 UTC m=+559.604047760" Jul 07 02:27:25.073857 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:25.073774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:27:28.793020 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:28.792986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-nk5gk" Jul 07 02:27:33.817774 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:33.817746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-4dsts" Jul 07 02:27:35.825666 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:35.825635 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-gd78l" Jul 07 02:27:55.680088 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:55.680057 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:27:55.681255 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:27:55.681228 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:28:20.466658 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.466560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:20.619748 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.619709 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 02:28:20.622934 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.622910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:20.627733 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.627706 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 07 02:28:20.628850 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.628819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-jtx9s\"" Jul 07 02:28:20.629025 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.628852 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 07 02:28:20.629025 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.628866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 07 02:28:20.645025 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.644995 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 02:28:20.767173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.767072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:20.767173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.767164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pzp\" (UniqueName: \"kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:20.868208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.868163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pzp\" (UniqueName: \"kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:20.868208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.868216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:20.868428 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:28:20.868320 2564 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 02:28:20.868428 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:28:20.868400 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert podName:8faafaff-3a50-4922-b221-48162e703d4d nodeName:}" failed. No retries permitted until 2026-07-07 02:28:21.368382852 +0000 UTC m=+626.125931856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert") pod "llmisvc-controller-manager-7fb88b8c57-q5hr2" (UID: "8faafaff-3a50-4922-b221-48162e703d4d") : secret "llmisvc-webhook-server-cert" not found Jul 07 02:28:20.879160 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:20.879125 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pzp\" (UniqueName: \"kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:21.373002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.372967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:21.375645 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.375609 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") pod \"llmisvc-controller-manager-7fb88b8c57-q5hr2\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:21.533167 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.533124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:21.700959 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.700933 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 02:28:21.703473 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:28:21.703446 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8faafaff_3a50_4922_b221_48162e703d4d.slice/crio-be1c4654cfc6308cd1e6fb3b57098661802e181716949d21c0e326312c7d26e3 WatchSource:0}: Error finding container be1c4654cfc6308cd1e6fb3b57098661802e181716949d21c0e326312c7d26e3: Status 404 returned error can't find the container with id be1c4654cfc6308cd1e6fb3b57098661802e181716949d21c0e326312c7d26e3 Jul 07 02:28:21.705183 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.705163 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:28:21.715120 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:21.715089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:22.072607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:22.072575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:22.090825 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:22.090766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" event={"ID":"8faafaff-3a50-4922-b221-48162e703d4d","Type":"ContainerStarted","Data":"be1c4654cfc6308cd1e6fb3b57098661802e181716949d21c0e326312c7d26e3"} Jul 07 02:28:22.205513 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:22.205468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:25.107600 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:25.107565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" event={"ID":"8faafaff-3a50-4922-b221-48162e703d4d","Type":"ContainerStarted","Data":"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8"} Jul 07 02:28:25.108103 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:25.107653 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:25.134536 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:25.134480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" podStartSLOduration=1.913467068 podStartE2EDuration="5.134463073s" totalImagesPullingTime="3.220996005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:28:25.133215169 +0000 UTC m=+629.890764222" watchObservedRunningTime="2026-07-07 02:28:25.134463073 +0000 UTC m=+629.892012098" Jul 07 02:28:26.696227 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:26.696194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:26.807494 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:26.807451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:43.654630 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:43.654592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:56.113890 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:56.113850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 02:28:57.503434 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.503397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:57.604528 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.604489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:28:57.783861 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.783814 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w"] Jul 07 02:28:57.787166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.787131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.790213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.790182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 02:28:57.790213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.790182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 02:28:57.791414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.791389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rqcdc\"" Jul 07 02:28:57.803559 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.803517 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w"] Jul 07 02:28:57.860381 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.860344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.860580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.860391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.860580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.860412 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2qb\" (UniqueName: \"kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.961260 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.961222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.961260 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.961263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2qb\" (UniqueName: \"kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.961475 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.961318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.961719 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.961696 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.961760 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.961707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:57.971414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:57.971385 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2qb\" (UniqueName: \"kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:58.098488 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:58.098394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:28:58.259760 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:58.259711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w"] Jul 07 02:28:58.263665 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:28:58.263629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3105d1a_e8d1_4cbd_adcc_a2f8058b0d0f.slice/crio-560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08 WatchSource:0}: Error finding container 560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08: Status 404 returned error can't find the container with id 560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08 Jul 07 02:28:59.242600 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:59.242482 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerID="c4c45bd797671a8dcbb0958aaf879ad95ba2280636f6c86c5e112f06d9dcdb95" exitCode=0 Jul 07 02:28:59.242600 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:59.242584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerDied","Data":"c4c45bd797671a8dcbb0958aaf879ad95ba2280636f6c86c5e112f06d9dcdb95"} Jul 07 02:28:59.243056 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:28:59.242646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerStarted","Data":"560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08"} Jul 07 02:29:01.257755 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:01.257714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerStarted","Data":"572c70155d8f4c40ca59b3d4a6c15154344a812c6b32a1bff4e13f177607cea7"} Jul 07 02:29:02.263582 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:02.263534 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerID="572c70155d8f4c40ca59b3d4a6c15154344a812c6b32a1bff4e13f177607cea7" exitCode=0 Jul 07 02:29:02.264074 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:02.263620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerDied","Data":"572c70155d8f4c40ca59b3d4a6c15154344a812c6b32a1bff4e13f177607cea7"} Jul 07 02:29:03.270614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:03.270579 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerID="2c7a70417e6ac3c86441c2ce38c7a1cb556a0e6a81c04e80bda6c786d6806c64" exitCode=0 Jul 07 02:29:03.271060 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:03.270626 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerDied","Data":"2c7a70417e6ac3c86441c2ce38c7a1cb556a0e6a81c04e80bda6c786d6806c64"} Jul 07 02:29:04.408057 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.408014 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:29:04.515014 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.514970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util\") pod \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " Jul 07 02:29:04.515194 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.515028 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle\") pod \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " Jul 07 02:29:04.515194 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.515096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g2qb\" (UniqueName: \"kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb\") pod \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\" (UID: \"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f\") " Jul 07 02:29:04.515950 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.515849 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle" (OuterVolumeSpecName: "bundle") pod "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" (UID: "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:29:04.517459 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.517429 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb" (OuterVolumeSpecName: "kube-api-access-4g2qb") pod "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" (UID: "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f"). InnerVolumeSpecName "kube-api-access-4g2qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:29:04.521407 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.521366 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util" (OuterVolumeSpecName: "util") pod "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" (UID: "e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:29:04.615945 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.615895 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:04.615945 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.615934 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2qb\" (UniqueName: \"kubernetes.io/projected/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-kube-api-access-4g2qb\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:04.615945 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:04.615950 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:05.281746 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:05.281701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" event={"ID":"e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f","Type":"ContainerDied","Data":"560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08"} Jul 07 02:29:05.281958 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:05.281729 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kn92w" Jul 07 02:29:05.281958 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:05.281825 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="560a09dba08e1699f98ac8fbc9e677d4a49e8ab313c71cdb8dab5b8bce25ef08" Jul 07 02:29:08.713027 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:08.712984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:09.537536 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:09.537501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:12.482874 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:12.482838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:12.583257 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:12.583217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:14.820241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:14.820208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:14.912360 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:14.912310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:16.838487 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:16.838446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:17.230527 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:17.230428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:17.331264 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:17.331225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:18.964651 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:18.964610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:19.066933 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.066900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:19.168057 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.168013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:19.898100 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898063 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6"] Jul 07 02:29:19.898366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898353 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="extract" Jul 07 02:29:19.898415 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898367 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="extract" Jul 07 02:29:19.898415 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898379 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="util" Jul 07 02:29:19.898415 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898385 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="util" Jul 07 02:29:19.898415 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898394 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="pull" Jul 07 02:29:19.898415 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898400 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="pull" Jul 07 02:29:19.898565 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.898439 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3105d1a-e8d1-4cbd-adcc-a2f8058b0d0f" containerName="extract" Jul 07 02:29:19.901487 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.901461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:19.904212 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.904165 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 02:29:19.904212 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.904193 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rqcdc\"" Jul 07 02:29:19.905229 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.905211 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 02:29:19.911453 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.911410 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6"] Jul 07 02:29:19.943887 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.943838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:19.943887 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.943881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:19.944161 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:19.943911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x256k\" (UniqueName: \"kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.045175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.045128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.045175 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.045169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.045594 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.045202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x256k\" (UniqueName: \"kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.045594 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.045554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.045594 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.045574 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.053736 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.053694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x256k\" (UniqueName: \"kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.213168 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.213053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:20.356642 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:20.356600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6"] Jul 07 02:29:20.360890 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:29:20.360851 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8864515_8bee_43ed_8eb9_0f9bc98eed0d.slice/crio-c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8 WatchSource:0}: Error finding container c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8: Status 404 returned error can't find the container with id c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8 Jul 07 02:29:21.354000 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:21.353908 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerID="a8ed0dd484f8f4219d5b2dbebf6eab5f2b9bf3d1289eb1218dd3bab776add349" exitCode=0 Jul 07 02:29:21.354000 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:21.353948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" event={"ID":"d8864515-8bee-43ed-8eb9-0f9bc98eed0d","Type":"ContainerDied","Data":"a8ed0dd484f8f4219d5b2dbebf6eab5f2b9bf3d1289eb1218dd3bab776add349"} Jul 07 02:29:21.354000 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:21.353985 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" event={"ID":"d8864515-8bee-43ed-8eb9-0f9bc98eed0d","Type":"ContainerStarted","Data":"c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8"} Jul 07 02:29:25.425147 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:25.425105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:26.382877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:26.382839 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerID="43dfad3f2d4f9a6b4a48a63cd7a7e8f047c645b59dfdef8d8bf7288f427ca7a5" exitCode=0 Jul 07 02:29:26.383054 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:26.382887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" event={"ID":"d8864515-8bee-43ed-8eb9-0f9bc98eed0d","Type":"ContainerDied","Data":"43dfad3f2d4f9a6b4a48a63cd7a7e8f047c645b59dfdef8d8bf7288f427ca7a5"} Jul 07 02:29:27.389474 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:27.389435 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerID="0972690849ecb393364d9d0bade4b12ed277fdee4b0b2f6c35d1c063f2e9a121" exitCode=0 Jul 07 02:29:27.389932 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:27.389534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" event={"ID":"d8864515-8bee-43ed-8eb9-0f9bc98eed0d","Type":"ContainerDied","Data":"0972690849ecb393364d9d0bade4b12ed277fdee4b0b2f6c35d1c063f2e9a121"} Jul 07 02:29:28.530580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.530554 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:28.613865 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.613825 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x256k\" (UniqueName: \"kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k\") pod \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " Jul 07 02:29:28.614088 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.613897 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util\") pod \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " Jul 07 02:29:28.614088 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.613956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle\") pod \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\" (UID: \"d8864515-8bee-43ed-8eb9-0f9bc98eed0d\") " Jul 07 02:29:28.614332 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.614304 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle" (OuterVolumeSpecName: "bundle") pod "d8864515-8bee-43ed-8eb9-0f9bc98eed0d" (UID: "d8864515-8bee-43ed-8eb9-0f9bc98eed0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:29:28.616202 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.616175 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k" (OuterVolumeSpecName: "kube-api-access-x256k") pod "d8864515-8bee-43ed-8eb9-0f9bc98eed0d" (UID: "d8864515-8bee-43ed-8eb9-0f9bc98eed0d"). InnerVolumeSpecName "kube-api-access-x256k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:29:28.618638 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.618598 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util" (OuterVolumeSpecName: "util") pod "d8864515-8bee-43ed-8eb9-0f9bc98eed0d" (UID: "d8864515-8bee-43ed-8eb9-0f9bc98eed0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:29:28.715127 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.715028 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x256k\" (UniqueName: \"kubernetes.io/projected/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-kube-api-access-x256k\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:28.715127 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.715062 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:28.715127 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:28.715077 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d8864515-8bee-43ed-8eb9-0f9bc98eed0d-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:29:29.400695 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:29.400667 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" Jul 07 02:29:29.400896 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:29.400666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft77z6" event={"ID":"d8864515-8bee-43ed-8eb9-0f9bc98eed0d","Type":"ContainerDied","Data":"c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8"} Jul 07 02:29:29.400896 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:29.400807 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c0105cbdf0d4be5881f9d8700f1be85a9a96eed79f38f5cbb3b51164e3b5d8" Jul 07 02:29:31.408680 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.408644 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76476b5bdb-gh7mz"] Jul 07 02:29:31.409111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409094 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="extract" Jul 07 02:29:31.409157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409111 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="extract" Jul 07 02:29:31.409157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409130 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="pull" Jul 07 02:29:31.409157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409138 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="pull" Jul 07 02:29:31.409246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409163 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="util" Jul 07 02:29:31.409246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409172 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="util" Jul 07 02:29:31.409304 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.409251 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8864515-8bee-43ed-8eb9-0f9bc98eed0d" containerName="extract" Jul 07 02:29:31.415611 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.415582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.423641 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.423608 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76476b5bdb-gh7mz"] Jul 07 02:29:31.468502 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.468463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:31.538168 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538368 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qbm\" (UniqueName: \"kubernetes.io/projected/033074c9-4c42-4217-893d-d8cbebd392e4-kube-api-access-25qbm\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538368 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-console-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538368 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-service-ca\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538368 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-trusted-ca-bundle\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538368 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-oauth-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.538538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.538386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-oauth-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.569327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.569289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:31.639803 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qbm\" (UniqueName: \"kubernetes.io/projected/033074c9-4c42-4217-893d-d8cbebd392e4-kube-api-access-25qbm\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-console-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-service-ca\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-trusted-ca-bundle\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-oauth-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.639970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-oauth-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640823 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.640765 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-oauth-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640950 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.640881 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-service-ca\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.640966 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-trusted-ca-bundle\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.640995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.640970 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/033074c9-4c42-4217-893d-d8cbebd392e4-console-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.642504 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.642477 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-serving-cert\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.642588 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.642513 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/033074c9-4c42-4217-893d-d8cbebd392e4-console-oauth-config\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.649942 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.649909 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qbm\" (UniqueName: \"kubernetes.io/projected/033074c9-4c42-4217-893d-d8cbebd392e4-kube-api-access-25qbm\") pod \"console-76476b5bdb-gh7mz\" (UID: \"033074c9-4c42-4217-893d-d8cbebd392e4\") " pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.725760 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.725664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:31.872125 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:31.872087 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76476b5bdb-gh7mz"] Jul 07 02:29:31.875223 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:29:31.875185 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033074c9_4c42_4217_893d_d8cbebd392e4.slice/crio-2ec0166fc877f61ee26520988218c0913b531fd6617e935aa5c57ce46d2b5e45 WatchSource:0}: Error finding container 2ec0166fc877f61ee26520988218c0913b531fd6617e935aa5c57ce46d2b5e45: Status 404 returned error can't find the container with id 2ec0166fc877f61ee26520988218c0913b531fd6617e935aa5c57ce46d2b5e45 Jul 07 02:29:32.399150 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:32.399112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:32.416812 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:32.416760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76476b5bdb-gh7mz" event={"ID":"033074c9-4c42-4217-893d-d8cbebd392e4","Type":"ContainerStarted","Data":"75c576af553f2429a26f69912c8074ebb5991c357527bba1fd2903780252068e"} Jul 07 02:29:32.417313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:32.416831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76476b5bdb-gh7mz" event={"ID":"033074c9-4c42-4217-893d-d8cbebd392e4","Type":"ContainerStarted","Data":"2ec0166fc877f61ee26520988218c0913b531fd6617e935aa5c57ce46d2b5e45"} Jul 07 02:29:32.439387 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:32.439317 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76476b5bdb-gh7mz" podStartSLOduration=1.439293795 podStartE2EDuration="1.439293795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:32.439181516 +0000 UTC m=+697.196730541" watchObservedRunningTime="2026-07-07 02:29:32.439293795 +0000 UTC m=+697.196842820" Jul 07 02:29:32.498393 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:32.498347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:41.725895 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:41.725838 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:41.726300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:41.725948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:41.731331 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:41.731305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:41.982729 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:41.982624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:42.050020 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.049978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:42.463304 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.463272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76476b5bdb-gh7mz" Jul 07 02:29:42.515486 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.515439 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:29:42.643386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.643351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:42.745192 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.745081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:42.847056 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.847016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:42.914113 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:42.914073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:47.896978 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.896841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:47.917405 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.917367 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl"] Jul 07 02:29:47.923065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.923038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:47.930281 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.930250 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 02:29:47.931937 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.931909 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 02:29:47.931937 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.931932 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 02:29:47.932166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.931909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 02:29:47.932166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.931917 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 02:29:47.932166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.932013 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 02:29:47.932166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.931910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gb4c7\"" Jul 07 02:29:47.942618 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:47.942573 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl"] Jul 07 02:29:48.007150 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.007111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:48.075051 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.074993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdrm\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-kube-api-access-8zdrm\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075256 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe608940-dc4c-464e-b0d7-6b8be3b52c99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.075454 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.075310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.176752 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.176655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.176752 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.176699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.176752 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.176721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdrm\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-kube-api-access-8zdrm\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.176989 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.176917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe608940-dc4c-464e-b0d7-6b8be3b52c99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.177022 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.176999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.177069 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.177051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.177148 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.177110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.177479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.177452 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.180370 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.180330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe608940-dc4c-464e-b0d7-6b8be3b52c99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.180370 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.180368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.180574 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.180368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.180574 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.180368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.189865 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.189830 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdrm\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-kube-api-access-8zdrm\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.191886 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.191858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe608940-dc4c-464e-b0d7-6b8be3b52c99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ngnzl\" (UID: \"fe608940-dc4c-464e-b0d7-6b8be3b52c99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.234505 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.234464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:48.395209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.395172 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl"] Jul 07 02:29:48.398623 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:29:48.398595 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe608940_dc4c_464e_b0d7_6b8be3b52c99.slice/crio-a18e1cbd2443f66c70fdd6f4819229f4cb38a855aa263664309aaaf6a5dd80e5 WatchSource:0}: Error finding container a18e1cbd2443f66c70fdd6f4819229f4cb38a855aa263664309aaaf6a5dd80e5: Status 404 returned error can't find the container with id a18e1cbd2443f66c70fdd6f4819229f4cb38a855aa263664309aaaf6a5dd80e5 Jul 07 02:29:48.489902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:48.489762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" event={"ID":"fe608940-dc4c-464e-b0d7-6b8be3b52c99","Type":"ContainerStarted","Data":"a18e1cbd2443f66c70fdd6f4819229f4cb38a855aa263664309aaaf6a5dd80e5"} Jul 07 02:29:49.159567 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:49.159527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:51.192426 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.192368 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:29:51.192772 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.192460 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:29:51.511674 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.511565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" event={"ID":"fe608940-dc4c-464e-b0d7-6b8be3b52c99","Type":"ContainerStarted","Data":"f3b173af03cb8cbf344ba0376d99e2bfef9ba7bff797fd6f600beb7c89d1c88a"} Jul 07 02:29:51.511880 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.511704 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:51.536429 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.536374 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" podStartSLOduration=1.744947631 podStartE2EDuration="4.536353705s" totalImagesPullingTime="2.791406074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:51.531857312 +0000 UTC m=+716.289406359" watchObservedRunningTime="2026-07-07 02:29:51.536353705 +0000 UTC m=+716.293902729" Jul 07 02:29:51.897496 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.897459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:51.978936 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:51.978896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:52.518575 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:52.518543 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-ngnzl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 02:29:52.518987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:52.518598 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" podUID="fe608940-dc4c-464e-b0d7-6b8be3b52c99" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 02:29:54.875622 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:54.875554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:54.966326 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:54.966292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:55.518157 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:55.518123 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ngnzl" Jul 07 02:29:56.027124 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.027084 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl"] Jul 07 02:29:56.029422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.029395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.034075 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.034051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-c9cfv\"" Jul 07 02:29:56.046355 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046355 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f81fc67-aedb-4b88-867c-7804aadffb05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzbq\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-kube-api-access-dkzbq\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046771 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046771 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.046771 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.046749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.051364 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.051326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl"] Jul 07 02:29:56.058596 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.058561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:56.141630 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.141596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:29:56.147720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147720 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f81fc67-aedb-4b88-867c-7804aadffb05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzbq\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-kube-api-access-dkzbq\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.147987 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.147953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.148336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.148227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.148336 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.148249 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.148550 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.148526 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.148612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.148556 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.148692 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.148675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f81fc67-aedb-4b88-867c-7804aadffb05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.150755 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.150729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.150890 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.150737 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.158146 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.158112 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.158304 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.158194 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzbq\" (UniqueName: \"kubernetes.io/projected/5f81fc67-aedb-4b88-867c-7804aadffb05-kube-api-access-dkzbq\") pod \"openshift-ai-inference-openshift-default-866675cbb8-c9ndl\" (UID: \"5f81fc67-aedb-4b88-867c-7804aadffb05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.342559 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.342521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:29:56.531419 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:56.531392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl"] Jul 07 02:29:56.534005 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:29:56.533969 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f81fc67_aedb_4b88_867c_7804aadffb05.slice/crio-c711789ddb4543648ee4f23d4f75d969bcd578069008e3a3b185f81a82580aed WatchSource:0}: Error finding container c711789ddb4543648ee4f23d4f75d969bcd578069008e3a3b185f81a82580aed: Status 404 returned error can't find the container with id c711789ddb4543648ee4f23d4f75d969bcd578069008e3a3b185f81a82580aed Jul 07 02:29:57.545250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:57.545209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerStarted","Data":"c711789ddb4543648ee4f23d4f75d969bcd578069008e3a3b185f81a82580aed"} Jul 07 02:29:59.693634 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:29:59.693591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:01.181573 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:01.181530 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:30:01.181853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:01.181630 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:30:01.181853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:01.181664 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:30:01.568450 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:01.568357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerStarted","Data":"465c738ceed819707e0d56d2d8ab05bb55be2ddf857721d6ac5609aa34f2e709"} Jul 07 02:30:01.593399 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:01.593342 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podStartSLOduration=1.9485285719999998 podStartE2EDuration="6.593324782s" totalImagesPullingTime="4.64479621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:01.591873841 +0000 UTC m=+726.349422885" watchObservedRunningTime="2026-07-07 02:30:01.593324782 +0000 UTC m=+726.350873807" Jul 07 02:30:02.343710 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.343665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:30:02.349031 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.348999 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:30:02.573308 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.573271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:30:02.574270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.574248 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:30:02.826408 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.826369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:02.921929 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:02.921828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:05.081241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.081192 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg"] Jul 07 02:30:05.084246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.084221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.087162 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.087135 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 02:30:05.087349 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.087189 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 02:30:05.088803 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.088766 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rqcdc\"" Jul 07 02:30:05.094241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.094205 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg"] Jul 07 02:30:05.128202 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.128158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8pw\" (UniqueName: \"kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.128392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.128210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.128392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.128255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.185141 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.185103 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt"] Jul 07 02:30:05.187579 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.187555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.198941 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.198907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt"] Jul 07 02:30:05.229401 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kc4\" (UniqueName: \"kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.229614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.229614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8pw\" (UniqueName: \"kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.229614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.229614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.229614 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.229582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.230024 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.230001 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.230125 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.230025 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.240259 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.240211 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8pw\" (UniqueName: \"kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.283025 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.282980 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m"] Jul 07 02:30:05.285689 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.285665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.297242 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.297204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m"] Jul 07 02:30:05.330499 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.330711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kc4\" (UniqueName: \"kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.330711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.330711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.330711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.330711 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.330667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhv27\" (UniqueName: \"kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.331080 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.331052 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.331120 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.331059 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.342433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.342346 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kc4\" (UniqueName: \"kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.388361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.388310 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq"] Jul 07 02:30:05.391122 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.391087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.395333 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.395302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:05.401298 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.401254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq"] Jul 07 02:30:05.431248 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.431248 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.431522 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdp2\" (UniqueName: \"kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.431522 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.431522 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhv27\" (UniqueName: \"kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.431678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.431730 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431679 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.431914 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.431894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.444003 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.443957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhv27\" (UniqueName: \"kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.498169 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.498125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:05.532852 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.532815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.533044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.532876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdp2\" (UniqueName: \"kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.533044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.532931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.533366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.533334 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.533446 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.533349 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.542351 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.542311 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdp2\" (UniqueName: \"kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.551517 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.551481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg"] Jul 07 02:30:05.553620 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:30:05.553580 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0405c04_d511_4d34_af6f_7d906ca3d3d8.slice/crio-6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b WatchSource:0}: Error finding container 6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b: Status 404 returned error can't find the container with id 6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b Jul 07 02:30:05.589592 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.589546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" event={"ID":"c0405c04-d511-4d34-af6f-7d906ca3d3d8","Type":"ContainerStarted","Data":"6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b"} Jul 07 02:30:05.597073 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.596856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:05.664624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.664403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt"] Jul 07 02:30:05.668694 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:30:05.668616 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c35043_7dff_4663_b1db_7a87ad0930c5.slice/crio-ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a WatchSource:0}: Error finding container ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a: Status 404 returned error can't find the container with id ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a Jul 07 02:30:05.703482 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.703447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:05.773744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.773715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m"] Jul 07 02:30:05.869263 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:05.869229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq"] Jul 07 02:30:05.884136 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:30:05.884104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992b2309_feea_4268_9594_c2277dddee39.slice/crio-48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1 WatchSource:0}: Error finding container 48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1: Status 404 returned error can't find the container with id 48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1 Jul 07 02:30:06.596115 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.596077 2564 generic.go:356] "Generic (PLEG): container finished" podID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerID="b3bbb5a645a908f530066bfdf1742fdce3a892ca19f079d4076b949df60099a2" exitCode=0 Jul 07 02:30:06.596572 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.596166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" event={"ID":"945af7b0-9f7d-49e5-aae5-85dd4fba6a18","Type":"ContainerDied","Data":"b3bbb5a645a908f530066bfdf1742fdce3a892ca19f079d4076b949df60099a2"} Jul 07 02:30:06.596572 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.596226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" event={"ID":"945af7b0-9f7d-49e5-aae5-85dd4fba6a18","Type":"ContainerStarted","Data":"ef9c022504bd3b4b1279c87d77ef5b8830de27a4d22ac4daa025fa886fb9afe1"} Jul 07 02:30:06.598086 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.598060 2564 generic.go:356] "Generic (PLEG): container finished" podID="992b2309-feea-4268-9594-c2277dddee39" containerID="65017958247deeb9f366d90ce2025dacbea30747f7e3d0a19ba013f17f1bc927" exitCode=0 Jul 07 02:30:06.598220 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.598142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" event={"ID":"992b2309-feea-4268-9594-c2277dddee39","Type":"ContainerDied","Data":"65017958247deeb9f366d90ce2025dacbea30747f7e3d0a19ba013f17f1bc927"} Jul 07 02:30:06.598220 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.598215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" event={"ID":"992b2309-feea-4268-9594-c2277dddee39","Type":"ContainerStarted","Data":"48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1"} Jul 07 02:30:06.600021 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.599979 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerID="8e167acec3c17295363e73b34d1ee84880c52eb8345e7484ad444404a23db721" exitCode=0 Jul 07 02:30:06.600145 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.600069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" event={"ID":"c0405c04-d511-4d34-af6f-7d906ca3d3d8","Type":"ContainerDied","Data":"8e167acec3c17295363e73b34d1ee84880c52eb8345e7484ad444404a23db721"} Jul 07 02:30:06.601992 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.601961 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerID="8023dc3823e9b045a19b3ec03d4981d4fcbc66f3baa97c72ea0f722c4f619a15" exitCode=0 Jul 07 02:30:06.602091 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.602007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" event={"ID":"a6c35043-7dff-4663-b1db-7a87ad0930c5","Type":"ContainerDied","Data":"8023dc3823e9b045a19b3ec03d4981d4fcbc66f3baa97c72ea0f722c4f619a15"} Jul 07 02:30:06.602091 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:06.602043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" event={"ID":"a6c35043-7dff-4663-b1db-7a87ad0930c5","Type":"ContainerStarted","Data":"ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a"} Jul 07 02:30:07.538934 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.538886 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7678dd94b4-gbcrn" podUID="281eb1a8-fe8b-4064-a308-9ffbc497951a" containerName="console" containerID="cri-o://7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc" gracePeriod=15 Jul 07 02:30:07.819039 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.819006 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:30:07.856272 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856272 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856278 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856308 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856336 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjcr2\" (UniqueName: \"kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert\") pod \"281eb1a8-fe8b-4064-a308-9ffbc497951a\" (UID: \"281eb1a8-fe8b-4064-a308-9ffbc497951a\") " Jul 07 02:30:07.856863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856723 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca" (OuterVolumeSpecName: "service-ca") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:30:07.856863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856742 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:30:07.856863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856849 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-trusted-ca-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.857009 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.856868 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-service-ca\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.857395 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.857197 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config" (OuterVolumeSpecName: "console-config") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:30:07.857395 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.857247 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:30:07.859100 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.858954 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:30:07.859293 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.859275 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:30:07.859534 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.859515 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2" (OuterVolumeSpecName: "kube-api-access-jjcr2") pod "281eb1a8-fe8b-4064-a308-9ffbc497951a" (UID: "281eb1a8-fe8b-4064-a308-9ffbc497951a"). InnerVolumeSpecName "kube-api-access-jjcr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:30:07.958065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.957910 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.958065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.957974 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjcr2\" (UniqueName: \"kubernetes.io/projected/281eb1a8-fe8b-4064-a308-9ffbc497951a-kube-api-access-jjcr2\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.958065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.957990 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.958065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.958003 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/281eb1a8-fe8b-4064-a308-9ffbc497951a-console-oauth-config\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:07.958065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:07.958016 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/281eb1a8-fe8b-4064-a308-9ffbc497951a-oauth-serving-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:08.634890 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.634848 2564 generic.go:356] "Generic (PLEG): container finished" podID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerID="17ab23c3d04f6b6e8e0ab37e39dd495d70257927ecd894eef54b539edc0cd464" exitCode=0 Jul 07 02:30:08.635079 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.634943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" event={"ID":"945af7b0-9f7d-49e5-aae5-85dd4fba6a18","Type":"ContainerDied","Data":"17ab23c3d04f6b6e8e0ab37e39dd495d70257927ecd894eef54b539edc0cd464"} Jul 07 02:30:08.637270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.637236 2564 generic.go:356] "Generic (PLEG): container finished" podID="281eb1a8-fe8b-4064-a308-9ffbc497951a" containerID="7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc" exitCode=0 Jul 07 02:30:08.637547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.637314 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7678dd94b4-gbcrn" Jul 07 02:30:08.637547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.637322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7678dd94b4-gbcrn" event={"ID":"281eb1a8-fe8b-4064-a308-9ffbc497951a","Type":"ContainerDied","Data":"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc"} Jul 07 02:30:08.637547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.637378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7678dd94b4-gbcrn" event={"ID":"281eb1a8-fe8b-4064-a308-9ffbc497951a","Type":"ContainerDied","Data":"6eeafd8fefc9f4932dd8cb4ef798fbdac104503016e522b78802f09831db14b4"} Jul 07 02:30:08.637547 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.637396 2564 scope.go:117] "RemoveContainer" containerID="7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc" Jul 07 02:30:08.639826 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.639771 2564 generic.go:356] "Generic (PLEG): container finished" podID="992b2309-feea-4268-9594-c2277dddee39" containerID="632ea401266b0529b20d709c1e667fe9479d14850f1176b62abce3cd215d816a" exitCode=0 Jul 07 02:30:08.639826 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.639816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" event={"ID":"992b2309-feea-4268-9594-c2277dddee39","Type":"ContainerDied","Data":"632ea401266b0529b20d709c1e667fe9479d14850f1176b62abce3cd215d816a"} Jul 07 02:30:08.642976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.642947 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerID="b9501c34e5bed326b6ea90ab7524c14dce6d7b4c6301f9be6e19210d5182ae9c" exitCode=0 Jul 07 02:30:08.643134 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.643016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" event={"ID":"c0405c04-d511-4d34-af6f-7d906ca3d3d8","Type":"ContainerDied","Data":"b9501c34e5bed326b6ea90ab7524c14dce6d7b4c6301f9be6e19210d5182ae9c"} Jul 07 02:30:08.653156 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.653134 2564 scope.go:117] "RemoveContainer" containerID="7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc" Jul 07 02:30:08.653506 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:30:08.653482 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc\": container with ID starting with 7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc not found: ID does not exist" containerID="7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc" Jul 07 02:30:08.653587 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.653513 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc"} err="failed to get container status \"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc\": rpc error: code = NotFound desc = could not find container \"7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc\": container with ID starting with 7807d85e235d5061630c5c34f8c58ef9d7b0bad1d902774ece66de5cd7237efc not found: ID does not exist" Jul 07 02:30:08.693799 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.693742 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:30:08.698265 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:08.698168 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7678dd94b4-gbcrn"] Jul 07 02:30:09.650640 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.650601 2564 generic.go:356] "Generic (PLEG): container finished" podID="992b2309-feea-4268-9594-c2277dddee39" containerID="6c0470276f1745a75c040fae796768e17c58caf587e66fd58f3dac32487e31de" exitCode=0 Jul 07 02:30:09.651106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.650685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" event={"ID":"992b2309-feea-4268-9594-c2277dddee39","Type":"ContainerDied","Data":"6c0470276f1745a75c040fae796768e17c58caf587e66fd58f3dac32487e31de"} Jul 07 02:30:09.652985 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.652956 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerID="ee987213fc4f3f92b5c5c8d386f149e71c3786247375d4f65ff54cbed0e0a5a6" exitCode=0 Jul 07 02:30:09.653127 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.653046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" event={"ID":"c0405c04-d511-4d34-af6f-7d906ca3d3d8","Type":"ContainerDied","Data":"ee987213fc4f3f92b5c5c8d386f149e71c3786247375d4f65ff54cbed0e0a5a6"} Jul 07 02:30:09.654999 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.654975 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerID="d58335304d2ca4c045407388f2c9b4f36fdf0d913535bad8d32110637b8d16a1" exitCode=0 Jul 07 02:30:09.655120 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.655057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" event={"ID":"a6c35043-7dff-4663-b1db-7a87ad0930c5","Type":"ContainerDied","Data":"d58335304d2ca4c045407388f2c9b4f36fdf0d913535bad8d32110637b8d16a1"} Jul 07 02:30:09.657437 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.657407 2564 generic.go:356] "Generic (PLEG): container finished" podID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerID="e0b88623bdc3070fd0a1b6db53628c015420ce00de495b288262a02b9bb4ff98" exitCode=0 Jul 07 02:30:09.657551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.657488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" event={"ID":"945af7b0-9f7d-49e5-aae5-85dd4fba6a18","Type":"ContainerDied","Data":"e0b88623bdc3070fd0a1b6db53628c015420ce00de495b288262a02b9bb4ff98"} Jul 07 02:30:09.792522 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:09.792472 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281eb1a8-fe8b-4064-a308-9ffbc497951a" path="/var/lib/kubelet/pods/281eb1a8-fe8b-4064-a308-9ffbc497951a/volumes" Jul 07 02:30:10.666392 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.666351 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerID="1ef9e241f7fba485c1d7458f977191e7a3100cfd460070cb3e82c713d756f8e7" exitCode=0 Jul 07 02:30:10.666853 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.666418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" event={"ID":"a6c35043-7dff-4663-b1db-7a87ad0930c5","Type":"ContainerDied","Data":"1ef9e241f7fba485c1d7458f977191e7a3100cfd460070cb3e82c713d756f8e7"} Jul 07 02:30:10.828204 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.828171 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:10.842576 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.842549 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:10.847130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.847100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:10.887125 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.887012 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhv27\" (UniqueName: \"kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27\") pod \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " Jul 07 02:30:10.887125 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.887105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle\") pod \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " Jul 07 02:30:10.887388 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.887146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util\") pod \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\" (UID: \"945af7b0-9f7d-49e5-aae5-85dd4fba6a18\") " Jul 07 02:30:10.888403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.888367 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle" (OuterVolumeSpecName: "bundle") pod "945af7b0-9f7d-49e5-aae5-85dd4fba6a18" (UID: "945af7b0-9f7d-49e5-aae5-85dd4fba6a18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:10.890094 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.890043 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27" (OuterVolumeSpecName: "kube-api-access-xhv27") pod "945af7b0-9f7d-49e5-aae5-85dd4fba6a18" (UID: "945af7b0-9f7d-49e5-aae5-85dd4fba6a18"). InnerVolumeSpecName "kube-api-access-xhv27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:30:10.894050 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.894005 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util" (OuterVolumeSpecName: "util") pod "945af7b0-9f7d-49e5-aae5-85dd4fba6a18" (UID: "945af7b0-9f7d-49e5-aae5-85dd4fba6a18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:10.988313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle\") pod \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " Jul 07 02:30:10.988313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988297 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq8pw\" (UniqueName: \"kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw\") pod \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " Jul 07 02:30:10.988548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988327 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util\") pod \"992b2309-feea-4268-9594-c2277dddee39\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " Jul 07 02:30:10.988548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle\") pod \"992b2309-feea-4268-9594-c2277dddee39\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " Jul 07 02:30:10.988548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npdp2\" (UniqueName: \"kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2\") pod \"992b2309-feea-4268-9594-c2277dddee39\" (UID: \"992b2309-feea-4268-9594-c2277dddee39\") " Jul 07 02:30:10.988548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util\") pod \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\" (UID: \"c0405c04-d511-4d34-af6f-7d906ca3d3d8\") " Jul 07 02:30:10.988738 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988645 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhv27\" (UniqueName: \"kubernetes.io/projected/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-kube-api-access-xhv27\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:10.988738 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988664 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:10.988738 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988679 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/945af7b0-9f7d-49e5-aae5-85dd4fba6a18-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:10.988914 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988869 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle" (OuterVolumeSpecName: "bundle") pod "c0405c04-d511-4d34-af6f-7d906ca3d3d8" (UID: "c0405c04-d511-4d34-af6f-7d906ca3d3d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:10.988975 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.988943 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle" (OuterVolumeSpecName: "bundle") pod "992b2309-feea-4268-9594-c2277dddee39" (UID: "992b2309-feea-4268-9594-c2277dddee39"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:10.990730 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.990703 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2" (OuterVolumeSpecName: "kube-api-access-npdp2") pod "992b2309-feea-4268-9594-c2277dddee39" (UID: "992b2309-feea-4268-9594-c2277dddee39"). InnerVolumeSpecName "kube-api-access-npdp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:30:10.990991 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.990963 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw" (OuterVolumeSpecName: "kube-api-access-zq8pw") pod "c0405c04-d511-4d34-af6f-7d906ca3d3d8" (UID: "c0405c04-d511-4d34-af6f-7d906ca3d3d8"). InnerVolumeSpecName "kube-api-access-zq8pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:30:10.994323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.994289 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util" (OuterVolumeSpecName: "util") pod "992b2309-feea-4268-9594-c2277dddee39" (UID: "992b2309-feea-4268-9594-c2277dddee39"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:10.994976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:10.994952 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util" (OuterVolumeSpecName: "util") pod "c0405c04-d511-4d34-af6f-7d906ca3d3d8" (UID: "c0405c04-d511-4d34-af6f-7d906ca3d3d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:11.090002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.089965 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8pw\" (UniqueName: \"kubernetes.io/projected/c0405c04-d511-4d34-af6f-7d906ca3d3d8-kube-api-access-zq8pw\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.090002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.089997 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.090002 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.090007 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/992b2309-feea-4268-9594-c2277dddee39-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.090239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.090015 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npdp2\" (UniqueName: \"kubernetes.io/projected/992b2309-feea-4268-9594-c2277dddee39-kube-api-access-npdp2\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.090239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.090025 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.090239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.090032 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0405c04-d511-4d34-af6f-7d906ca3d3d8-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.673249 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.673213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" Jul 07 02:30:11.673697 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.673221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76xxm6m" event={"ID":"945af7b0-9f7d-49e5-aae5-85dd4fba6a18","Type":"ContainerDied","Data":"ef9c022504bd3b4b1279c87d77ef5b8830de27a4d22ac4daa025fa886fb9afe1"} Jul 07 02:30:11.673697 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.673341 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9c022504bd3b4b1279c87d77ef5b8830de27a4d22ac4daa025fa886fb9afe1" Jul 07 02:30:11.675271 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.675235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" event={"ID":"992b2309-feea-4268-9594-c2277dddee39","Type":"ContainerDied","Data":"48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1"} Jul 07 02:30:11.675433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.675289 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36wxzdq" Jul 07 02:30:11.675433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.675295 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48fcc0a7db217d80caa5d7f072fdf721d9d1d7a6d070b4b1fc891f0ebe6cb7d1" Jul 07 02:30:11.677367 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.677335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" event={"ID":"c0405c04-d511-4d34-af6f-7d906ca3d3d8","Type":"ContainerDied","Data":"6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b"} Jul 07 02:30:11.677498 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.677390 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf5c10f8c028b80efdd038a7f289b5044b59e76a482b9836deef9c4ee831b2b" Jul 07 02:30:11.677568 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.677513 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efzvtlg" Jul 07 02:30:11.804747 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.804398 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:11.896339 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.896302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6kc4\" (UniqueName: \"kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4\") pod \"a6c35043-7dff-4663-b1db-7a87ad0930c5\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " Jul 07 02:30:11.896339 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.896344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle\") pod \"a6c35043-7dff-4663-b1db-7a87ad0930c5\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " Jul 07 02:30:11.896563 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.896386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util\") pod \"a6c35043-7dff-4663-b1db-7a87ad0930c5\" (UID: \"a6c35043-7dff-4663-b1db-7a87ad0930c5\") " Jul 07 02:30:11.897014 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.896986 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle" (OuterVolumeSpecName: "bundle") pod "a6c35043-7dff-4663-b1db-7a87ad0930c5" (UID: "a6c35043-7dff-4663-b1db-7a87ad0930c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:11.898594 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.898568 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4" (OuterVolumeSpecName: "kube-api-access-p6kc4") pod "a6c35043-7dff-4663-b1db-7a87ad0930c5" (UID: "a6c35043-7dff-4663-b1db-7a87ad0930c5"). InnerVolumeSpecName "kube-api-access-p6kc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:30:11.902956 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.902911 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util" (OuterVolumeSpecName: "util") pod "a6c35043-7dff-4663-b1db-7a87ad0930c5" (UID: "a6c35043-7dff-4663-b1db-7a87ad0930c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:30:11.997952 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.997863 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kc4\" (UniqueName: \"kubernetes.io/projected/a6c35043-7dff-4663-b1db-7a87ad0930c5-kube-api-access-p6kc4\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.997952 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.997893 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-bundle\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:11.997952 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:11.997902 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6c35043-7dff-4663-b1db-7a87ad0930c5-util\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:30:12.684307 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:12.684276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" Jul 07 02:30:12.684686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:12.684304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcab4dqnt" event={"ID":"a6c35043-7dff-4663-b1db-7a87ad0930c5","Type":"ContainerDied","Data":"ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a"} Jul 07 02:30:12.684686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:12.684354 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec4044fb52b716ba98b2f0ffaf5bb68a6de0d0a780c178ea2f45959857fa084a" Jul 07 02:30:18.413238 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:18.413195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:18.475980 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:18.475940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:21.409587 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:21.409544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:23.453043 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.452982 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd"] Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453457 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281eb1a8-fe8b-4064-a308-9ffbc497951a" containerName="console" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453473 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="281eb1a8-fe8b-4064-a308-9ffbc497951a" containerName="console" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453489 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453497 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453509 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453517 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453530 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453537 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453553 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453562 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453569 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453577 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="extract" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453587 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453595 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453602 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453609 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453629 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453640 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453648 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453657 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453668 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453675 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453686 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453693 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="util" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453702 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="pull" Jul 07 02:30:23.453743 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453709 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="pull" Jul 07 02:30:23.455383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453800 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="281eb1a8-fe8b-4064-a308-9ffbc497951a" containerName="console" Jul 07 02:30:23.455383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453810 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="945af7b0-9f7d-49e5-aae5-85dd4fba6a18" containerName="extract" Jul 07 02:30:23.455383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453820 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="992b2309-feea-4268-9594-c2277dddee39" containerName="extract" Jul 07 02:30:23.455383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453829 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c35043-7dff-4663-b1db-7a87ad0930c5" containerName="extract" Jul 07 02:30:23.455383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.453838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0405c04-d511-4d34-af6f-7d906ca3d3d8" containerName="extract" Jul 07 02:30:23.461123 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.461094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:23.465775 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.465748 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9qmhx\"" Jul 07 02:30:23.466811 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.466758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 02:30:23.467033 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.467014 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 02:30:23.470137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.470107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd"] Jul 07 02:30:23.598128 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.598085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xwn\" (UniqueName: \"kubernetes.io/projected/8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa-kube-api-access-t6xwn\") pod \"limitador-operator-controller-manager-6f9f468797-4wdtd\" (UID: \"8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:23.699383 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.699335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xwn\" (UniqueName: \"kubernetes.io/projected/8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa-kube-api-access-t6xwn\") pod \"limitador-operator-controller-manager-6f9f468797-4wdtd\" (UID: \"8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:23.714687 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.714612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xwn\" (UniqueName: \"kubernetes.io/projected/8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa-kube-api-access-t6xwn\") pod \"limitador-operator-controller-manager-6f9f468797-4wdtd\" (UID: \"8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa\") " pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:23.775406 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.775365 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:23.920470 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.920441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd"] Jul 07 02:30:23.922477 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:30:23.922437 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7ecf9b_60f5_4d8b_873c_731d5f70ebfa.slice/crio-6de76db04703fc2dad02e108483838eb1ff8901bc515153696b05d3a40015b4c WatchSource:0}: Error finding container 6de76db04703fc2dad02e108483838eb1ff8901bc515153696b05d3a40015b4c: Status 404 returned error can't find the container with id 6de76db04703fc2dad02e108483838eb1ff8901bc515153696b05d3a40015b4c Jul 07 02:30:23.992694 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:23.992600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:24.461160 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:24.461125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:24.748986 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:24.748727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" event={"ID":"8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa","Type":"ContainerStarted","Data":"6de76db04703fc2dad02e108483838eb1ff8901bc515153696b05d3a40015b4c"} Jul 07 02:30:24.823081 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:24.823046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:26.762597 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:26.762553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" event={"ID":"8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa","Type":"ContainerStarted","Data":"8e673d5e688b7f5436c9bf0f35df0d3e31c1a1ca384039cc0bd63b08ef64aa5a"} Jul 07 02:30:26.763031 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:26.762699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:26.782101 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:26.781982 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" podStartSLOduration=1.535487164 podStartE2EDuration="3.781962889s" totalImagesPullingTime="2.246475725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:26.781848652 +0000 UTC m=+751.539397678" watchObservedRunningTime="2026-07-07 02:30:26.781962889 +0000 UTC m=+751.539511915" Jul 07 02:30:27.252580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.252546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:27.612289 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.612248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj"] Jul 07 02:30:27.615898 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.615867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.619240 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.619208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-68d96\"" Jul 07 02:30:27.629809 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.629753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj"] Jul 07 02:30:27.733366 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.733320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/917bc873-ba60-4dd9-b14d-52eaac877e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.733560 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.733376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qts2k\" (UniqueName: \"kubernetes.io/projected/917bc873-ba60-4dd9-b14d-52eaac877e60-kube-api-access-qts2k\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.834037 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.834003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/917bc873-ba60-4dd9-b14d-52eaac877e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.834037 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.834045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qts2k\" (UniqueName: \"kubernetes.io/projected/917bc873-ba60-4dd9-b14d-52eaac877e60-kube-api-access-qts2k\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.834467 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.834411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/917bc873-ba60-4dd9-b14d-52eaac877e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.847379 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.847334 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qts2k\" (UniqueName: \"kubernetes.io/projected/917bc873-ba60-4dd9-b14d-52eaac877e60-kube-api-access-qts2k\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj\" (UID: \"917bc873-ba60-4dd9-b14d-52eaac877e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:27.926219 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:27.926127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:28.095320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:28.095295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj"] Jul 07 02:30:28.098195 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:30:28.098157 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917bc873_ba60_4dd9_b14d_52eaac877e60.slice/crio-e12e86bc7834d77a8ed865c8143afb7af7c416155c4fcf905a21190ce31a54fa WatchSource:0}: Error finding container e12e86bc7834d77a8ed865c8143afb7af7c416155c4fcf905a21190ce31a54fa: Status 404 returned error can't find the container with id e12e86bc7834d77a8ed865c8143afb7af7c416155c4fcf905a21190ce31a54fa Jul 07 02:30:28.302082 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:28.302052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:28.412365 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:28.412319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:28.516411 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:28.516370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:28.774435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:28.774340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" event={"ID":"917bc873-ba60-4dd9-b14d-52eaac877e60","Type":"ContainerStarted","Data":"e12e86bc7834d77a8ed865c8143afb7af7c416155c4fcf905a21190ce31a54fa"} Jul 07 02:30:30.218856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:30.218813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:30.476635 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:30.476552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:30.612159 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:30.612118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:33.046722 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:33.046680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:33.151766 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:33.151725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:34.209632 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:34.209599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:34.310345 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:34.310308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:35.816218 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:35.816175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" event={"ID":"917bc873-ba60-4dd9-b14d-52eaac877e60","Type":"ContainerStarted","Data":"bf133983b0e6cc20b2506328baa26b01f47b74ff2a938359fc956ec389372f0f"} Jul 07 02:30:35.816620 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:35.816398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:35.838147 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:35.838031 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" podStartSLOduration=1.355937282 podStartE2EDuration="8.838014491s" totalImagesPullingTime="7.482077209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:35.837400531 +0000 UTC m=+760.594949576" watchObservedRunningTime="2026-07-07 02:30:35.838014491 +0000 UTC m=+760.595563517" Jul 07 02:30:37.769848 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:37.769811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6f9f468797-4wdtd" Jul 07 02:30:38.109297 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:38.109253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:38.254386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:38.254346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:38.347133 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:38.347095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:38.601557 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:38.601513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:38.701592 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:38.701543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:39.016098 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:39.015965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:39.120608 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:39.120565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:44.471110 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:44.471069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:44.537879 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:44.537837 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:46.824216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:46.824187 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj" Jul 07 02:30:49.456440 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:49.456388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:49.557363 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:49.557324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:51.029956 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:51.029910 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:51.139062 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:51.139028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:53.532448 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:53.532404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:53.642953 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:53.642899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:56.890665 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:56.890617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:56.986491 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:56.986444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:30:57.087643 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:30:57.087590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:03.365750 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:03.365715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:03.465577 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:03.465536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:09.922503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:09.922468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:10.024295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:10.024258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:12.518621 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:12.518576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:12.620274 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:12.620235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:16.326131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:16.326035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:16.422766 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:16.422725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:16.534768 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:16.534728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:16.635221 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:16.635132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:17.529171 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:17.529139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:17.600224 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:17.600185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:20.543799 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.543742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:20.659649 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.659610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:20.730848 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.730811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:20.977943 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.977855 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-z5gcf"] Jul 07 02:31:20.980323 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.980284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:20.983096 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.983072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 02:31:20.983246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.983074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvhdn\"" Jul 07 02:31:20.994875 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:20.994839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-z5gcf"] Jul 07 02:31:21.071379 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.071339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-z5gcf"] Jul 07 02:31:21.102113 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.102075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f0719cf-edcf-4f8f-b606-511d32185922-config-file\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.102113 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.102117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flwq\" (UniqueName: \"kubernetes.io/projected/2f0719cf-edcf-4f8f-b606-511d32185922-kube-api-access-8flwq\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.203274 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.203236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f0719cf-edcf-4f8f-b606-511d32185922-config-file\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.203274 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.203278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flwq\" (UniqueName: \"kubernetes.io/projected/2f0719cf-edcf-4f8f-b606-511d32185922-kube-api-access-8flwq\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.203978 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.203957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f0719cf-edcf-4f8f-b606-511d32185922-config-file\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.212668 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.212625 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flwq\" (UniqueName: \"kubernetes.io/projected/2f0719cf-edcf-4f8f-b606-511d32185922-kube-api-access-8flwq\") pod \"limitador-limitador-69574b596d-z5gcf\" (UID: \"2f0719cf-edcf-4f8f-b606-511d32185922\") " pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.291625 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.291591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:21.443903 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.443874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-z5gcf"] Jul 07 02:31:21.445972 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:31:21.445940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0719cf_edcf_4f8f_b606_511d32185922.slice/crio-4ba0249024206449fd47a2686c3d95d39e6a71fafda163dd3427d275346bafa2 WatchSource:0}: Error finding container 4ba0249024206449fd47a2686c3d95d39e6a71fafda163dd3427d275346bafa2: Status 404 returned error can't find the container with id 4ba0249024206449fd47a2686c3d95d39e6a71fafda163dd3427d275346bafa2 Jul 07 02:31:21.468748 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.468710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:21.569185 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:21.569083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:22.047064 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:22.047029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" event={"ID":"2f0719cf-edcf-4f8f-b606-511d32185922","Type":"ContainerStarted","Data":"4ba0249024206449fd47a2686c3d95d39e6a71fafda163dd3427d275346bafa2"} Jul 07 02:31:24.384848 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:24.384808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:24.485433 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:24.485390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:26.076855 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:26.076813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" event={"ID":"2f0719cf-edcf-4f8f-b606-511d32185922","Type":"ContainerStarted","Data":"870f8fbbe086d53768b53b103249d36644313cf22b2e9a4c996461ab7a37b3c4"} Jul 07 02:31:26.077270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:26.077022 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:26.100933 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:26.100871 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" podStartSLOduration=2.236321735 podStartE2EDuration="6.1008533s" totalImagesPullingTime="3.864531565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:26.098638874 +0000 UTC m=+810.856187923" watchObservedRunningTime="2026-07-07 02:31:26.1008533 +0000 UTC m=+810.858402378" Jul 07 02:31:30.753387 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:30.753346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:30.936185 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:30.936133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:31.043250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:31.043207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:37.082852 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:37.082821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-z5gcf" Jul 07 02:31:43.173305 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:43.173267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:43.374516 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:43.374475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:57.509963 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:57.509927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:31:57.831929 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:31:57.831888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.008541 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.008495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.102173 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.102129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.253285 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.253248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.506648 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.506602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.683509 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.683481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:26.737747 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:26.737713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:29.347273 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:29.347236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:32:55.710746 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:55.710664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:32:55.713489 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:32:55.713460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:34:02.439910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:02.439861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:03.744449 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:03.744412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:14.688418 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.688344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:14.690856 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.690829 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.695267 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.695217 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:34:14.695462 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.695295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\"" Jul 07 02:34:14.696016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.695997 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" Jul 07 02:34:14.696115 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.696031 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:34:14.708413 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.708376 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:14.729103 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.729061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:14.797322 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.796944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797555 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lct\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797555 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797555 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797727 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797727 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797727 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797923 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.797923 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.797823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.835580 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.835521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:14.898641 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.898641 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.898957 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.898957 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.898957 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.898957 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.898988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lct\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899105 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899127 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899475 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899475 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899384 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.899973 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.899953 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.901593 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.901566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.902012 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.901988 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.916590 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.916548 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:14.916887 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:14.916858 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lct\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct\") pod \"router-gateway-1-openshift-default-799f46c59b-5qb88\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:15.004170 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.004057 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:15.165185 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.165140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:15.165382 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:34:15.165326 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03155af_70dd_47b1_a727_96e21071c514.slice/crio-4cbfa8f590c36b5b42079059893803f046851c83e52034bbef20e886367cac10 WatchSource:0}: Error finding container 4cbfa8f590c36b5b42079059893803f046851c83e52034bbef20e886367cac10: Status 404 returned error can't find the container with id 4cbfa8f590c36b5b42079059893803f046851c83e52034bbef20e886367cac10 Jul 07 02:34:15.167824 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.167773 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:15.167953 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.167859 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:15.167953 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.167888 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:15.950964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.950877 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" event={"ID":"a03155af-70dd-47b1-a727-96e21071c514","Type":"ContainerStarted","Data":"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f"} Jul 07 02:34:15.950964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.950929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" event={"ID":"a03155af-70dd-47b1-a727-96e21071c514","Type":"ContainerStarted","Data":"4cbfa8f590c36b5b42079059893803f046851c83e52034bbef20e886367cac10"} Jul 07 02:34:15.973467 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:15.973395 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" podStartSLOduration=1.9733756599999999 podStartE2EDuration="1.97337566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:15.971594956 +0000 UTC m=+980.729144004" watchObservedRunningTime="2026-07-07 02:34:15.97337566 +0000 UTC m=+980.730924686" Jul 07 02:34:16.004902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:16.004862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:16.010744 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:16.010711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:16.956065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:16.956029 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:16.957197 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:16.957168 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:17.757652 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:17.757581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:20.166886 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:20.166838 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:20.167442 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:20.167050 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" podUID="a03155af-70dd-47b1-a727-96e21071c514" containerName="istio-proxy" containerID="cri-o://62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f" gracePeriod=30 Jul 07 02:34:25.429546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.429456 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:25.597067 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597024 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597161 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597188 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6lct\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597270 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597328 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597354 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket\") pod \"a03155af-70dd-47b1-a727-96e21071c514\" (UID: \"a03155af-70dd-47b1-a727-96e21071c514\") " Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597464 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597487 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data" (OuterVolumeSpecName: "istio-data") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:34:25.597532 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597518 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:34:25.597877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597738 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:34:25.597877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597673 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03155af-70dd-47b1-a727-96e21071c514-istiod-ca-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.597877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597761 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:34:25.597877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597777 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-credential-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.597877 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.597826 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-data\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.600113 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.600064 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 02:34:25.600113 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.600074 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token" (OuterVolumeSpecName: "istio-token") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:34:25.600327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.600114 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct" (OuterVolumeSpecName: "kube-api-access-c6lct") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "kube-api-access-c6lct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:34:25.600327 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.600115 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a03155af-70dd-47b1-a727-96e21071c514" (UID: "a03155af-70dd-47b1-a727-96e21071c514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698672 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lct\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-kube-api-access-c6lct\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698712 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-istio-envoy\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698726 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698740 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03155af-70dd-47b1-a727-96e21071c514-istio-podinfo\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698751 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03155af-70dd-47b1-a727-96e21071c514-workload-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:25.698821 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:25.698762 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03155af-70dd-47b1-a727-96e21071c514-istio-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:34:26.007732 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.007621 2564 generic.go:356] "Generic (PLEG): container finished" podID="a03155af-70dd-47b1-a727-96e21071c514" containerID="62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f" exitCode=0 Jul 07 02:34:26.007732 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.007677 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" Jul 07 02:34:26.007732 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.007711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" event={"ID":"a03155af-70dd-47b1-a727-96e21071c514","Type":"ContainerDied","Data":"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f"} Jul 07 02:34:26.008046 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.007768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88" event={"ID":"a03155af-70dd-47b1-a727-96e21071c514","Type":"ContainerDied","Data":"4cbfa8f590c36b5b42079059893803f046851c83e52034bbef20e886367cac10"} Jul 07 02:34:26.008046 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.007815 2564 scope.go:117] "RemoveContainer" containerID="62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f" Jul 07 02:34:26.017313 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.017292 2564 scope.go:117] "RemoveContainer" containerID="62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f" Jul 07 02:34:26.017654 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:34:26.017630 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f\": container with ID starting with 62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f not found: ID does not exist" containerID="62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f" Jul 07 02:34:26.017714 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.017662 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f"} err="failed to get container status \"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f\": rpc error: code = NotFound desc = could not find container \"62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f\": container with ID starting with 62485a60a6bf268ec4e25a1499660840c4a56594e1fcb1470cb2e688fe8ac74f not found: ID does not exist" Jul 07 02:34:26.028403 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.028356 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:26.031455 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:26.031415 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-5qb88"] Jul 07 02:34:27.791757 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:27.791714 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03155af-70dd-47b1-a727-96e21071c514" path="/var/lib/kubelet/pods/a03155af-70dd-47b1-a727-96e21071c514/volumes" Jul 07 02:34:28.285260 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:28.285228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:30.095137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.095096 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:34:30.095607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.095468 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03155af-70dd-47b1-a727-96e21071c514" containerName="istio-proxy" Jul 07 02:34:30.095607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.095486 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03155af-70dd-47b1-a727-96e21071c514" containerName="istio-proxy" Jul 07 02:34:30.095607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.095538 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03155af-70dd-47b1-a727-96e21071c514" containerName="istio-proxy" Jul 07 02:34:30.098746 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.098714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.101825 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.101768 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:34:30.102005 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.101771 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:34:30.102005 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.101838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" Jul 07 02:34:30.102005 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.101993 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\"" Jul 07 02:34:30.113013 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.112975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:34:30.132431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132612 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132802 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132802 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.132802 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.132708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgprz\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.233967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.233924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.233967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.233970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.233997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgprz\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234570 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234453 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234570 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234512 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234675 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234647 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.234734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234718 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.235011 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.234989 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.236820 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.236769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.236967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.236945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.243303 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.243271 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.243473 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.243388 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgprz\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz\") pod \"router-gateway-1-openshift-default-799f46c59b-m74kj\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.412615 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.412508 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:30.568488 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.568452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:34:30.569838 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:34:30.569777 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301c6a18_a6ef_440d_a19f_764402eb980c.slice/crio-0d276eed17c26ca72c7e1765cb73827310f76713015292718362517b36670ec4 WatchSource:0}: Error finding container 0d276eed17c26ca72c7e1765cb73827310f76713015292718362517b36670ec4: Status 404 returned error can't find the container with id 0d276eed17c26ca72c7e1765cb73827310f76713015292718362517b36670ec4 Jul 07 02:34:30.572105 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.572045 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:30.572213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.572163 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:30.572213 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.572205 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:30.607605 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:30.607562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:31.036686 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:31.036647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerStarted","Data":"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9"} Jul 07 02:34:31.036913 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:31.036711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerStarted","Data":"0d276eed17c26ca72c7e1765cb73827310f76713015292718362517b36670ec4"} Jul 07 02:34:31.063678 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:31.063610 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" podStartSLOduration=1.063590426 podStartE2EDuration="1.063590426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:31.061752901 +0000 UTC m=+995.819301943" watchObservedRunningTime="2026-07-07 02:34:31.063590426 +0000 UTC m=+995.821139450" Jul 07 02:34:31.412907 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:31.412769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:31.418071 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:31.418041 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:32.042116 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:32.042091 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:32.043223 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:32.043196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:37.600810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:37.600431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:37.967217 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:37.967101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:41.141234 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:41.141188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:44.142820 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.142727 2564 generic.go:356] "Generic (PLEG): container finished" podID="301c6a18-a6ef-440d-a19f-764402eb980c" containerID="5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9" exitCode=0 Jul 07 02:34:44.142820 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.142808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerDied","Data":"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9"} Jul 07 02:34:44.143326 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.143213 2564 scope.go:117] "RemoveContainer" containerID="5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9" Jul 07 02:34:44.144220 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.144190 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:44.144277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.144266 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:44.144317 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:44.144302 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:45.151663 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:45.151629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerStarted","Data":"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942"} Jul 07 02:34:45.413804 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:45.413747 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:45.419180 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:45.419148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:46.158158 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:46.158105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:46.162837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:46.159213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:34:48.183557 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:48.183505 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": read tcp 10.133.0.2:37894->10.133.0.27:15021: read: connection reset by peer" start-of-body= Jul 07 02:34:48.184205 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:48.183605 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": read tcp 10.133.0.2:37894->10.133.0.27:15021: read: connection reset by peer" Jul 07 02:34:49.175624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.175586 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerID="465c738ceed819707e0d56d2d8ab05bb55be2ddf857721d6ac5609aa34f2e709" exitCode=0 Jul 07 02:34:49.175871 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.175639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerDied","Data":"465c738ceed819707e0d56d2d8ab05bb55be2ddf857721d6ac5609aa34f2e709"} Jul 07 02:34:49.176168 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.176146 2564 scope.go:117] "RemoveContainer" containerID="465c738ceed819707e0d56d2d8ab05bb55be2ddf857721d6ac5609aa34f2e709" Jul 07 02:34:49.177228 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.177192 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:49.177287 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.177275 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:49.177333 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:49.177321 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:34:50.187453 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:50.187405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerStarted","Data":"744f24a3f8fd8592de7875db472cfca133a9b72ef59648a3328bf1df43bb45e5"} Jul 07 02:34:50.343456 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:50.343408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:34:50.343704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:50.343684 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:50.343812 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:50.343732 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:51.179103 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:51.179054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:51.279906 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:51.279869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:34:51.343037 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:51.343001 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:51.343222 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:51.343058 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:52.343007 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:52.342959 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:52.343431 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:52.343022 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:53.343295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:53.343251 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:53.343721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:53.343311 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:54.344093 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:54.344044 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:54.344508 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:54.344127 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:55.343917 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:55.343860 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 02:34:55.344166 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:55.343923 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 02:34:56.347353 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:56.347311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:34:56.347825 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:56.347809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:34:56.348878 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:34:56.348856 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 02:35:00.957933 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.957890 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:00.961108 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.961082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:00.965126 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.965097 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 02:35:00.965321 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.965181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Jul 07 02:35:00.972628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.972590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:00.979464 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:00.979425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:01.019205 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.019205 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.019435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.019435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.019435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.019435 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.019393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rh8n\" (UniqueName: \"kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.120852 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.120803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.120886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.120908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rh8n\" (UniqueName: \"kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.120958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.120975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121055 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.121003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121325 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.121299 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121400 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.121377 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.121438 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.121398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.123736 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.123694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.123931 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.123892 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.130719 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.130676 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rh8n\" (UniqueName: \"kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.273309 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.273202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:01.437542 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.437495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:01.439235 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:35:01.439201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87801fb_0437_4fe2_a120_94a3f2a4bc6d.slice/crio-fe8033de3572e0fd0db15a0b0c82fe86b550cb013f013efbfe5aab4fa4003f43 WatchSource:0}: Error finding container fe8033de3572e0fd0db15a0b0c82fe86b550cb013f013efbfe5aab4fa4003f43: Status 404 returned error can't find the container with id fe8033de3572e0fd0db15a0b0c82fe86b550cb013f013efbfe5aab4fa4003f43 Jul 07 02:35:01.441405 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:01.441381 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 02:35:02.260692 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:02.260649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" event={"ID":"f87801fb-0437-4fe2-a120-94a3f2a4bc6d","Type":"ContainerStarted","Data":"fe8033de3572e0fd0db15a0b0c82fe86b550cb013f013efbfe5aab4fa4003f43"} Jul 07 02:35:03.268247 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:03.268205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" event={"ID":"f87801fb-0437-4fe2-a120-94a3f2a4bc6d","Type":"ContainerStarted","Data":"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336"} Jul 07 02:35:03.274124 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:03.274082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:03.288592 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:03.288509 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" podStartSLOduration=2.024999692 podStartE2EDuration="3.288481506s" totalImagesPullingTime="1.263481814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:35:03.28574981 +0000 UTC m=+1028.043298871" watchObservedRunningTime="2026-07-07 02:35:03.288481506 +0000 UTC m=+1028.046030532" Jul 07 02:35:11.274179 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:11.274136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:11.287228 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:11.287194 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:11.336703 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:11.336668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:11.987713 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:11.987674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:35:11.988931 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:11.987967 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" containerID="cri-o://49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942" gracePeriod=30 Jul 07 02:35:12.250417 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.250325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:35:12.323923 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.323888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.323956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.323987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324033 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324116 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgprz\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324183 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324233 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data\") pod \"301c6a18-a6ef-440d-a19f-764402eb980c\" (UID: \"301c6a18-a6ef-440d-a19f-764402eb980c\") " Jul 07 02:35:12.324445 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324348 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:12.324932 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324561 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data" (OuterVolumeSpecName: "istio-data") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:12.324932 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324572 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.324932 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324809 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:12.325080 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.324841 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:12.325430 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.325404 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:35:12.327152 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327089 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 02:35:12.327152 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327092 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:12.327152 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327121 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz" (OuterVolumeSpecName: "kube-api-access-xgprz") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "kube-api-access-xgprz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327200 2564 generic.go:356] "Generic (PLEG): container finished" podID="301c6a18-a6ef-440d-a19f-764402eb980c" containerID="49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942" exitCode=0 Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327235 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token" (OuterVolumeSpecName: "istio-token") pod "301c6a18-a6ef-440d-a19f-764402eb980c" (UID: "301c6a18-a6ef-440d-a19f-764402eb980c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327258 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerDied","Data":"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942"} Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj" event={"ID":"301c6a18-a6ef-440d-a19f-764402eb980c","Type":"ContainerDied","Data":"0d276eed17c26ca72c7e1765cb73827310f76713015292718362517b36670ec4"} Jul 07 02:35:12.327422 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.327424 2564 scope.go:117] "RemoveContainer" containerID="49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942" Jul 07 02:35:12.346115 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.346085 2564 scope.go:117] "RemoveContainer" containerID="5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9" Jul 07 02:35:12.355976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.355940 2564 scope.go:117] "RemoveContainer" containerID="49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942" Jul 07 02:35:12.356507 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:35:12.356475 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942\": container with ID starting with 49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942 not found: ID does not exist" containerID="49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942" Jul 07 02:35:12.356903 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.356516 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942"} err="failed to get container status \"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942\": rpc error: code = NotFound desc = could not find container \"49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942\": container with ID starting with 49b8c393f0783509583cba67b34912239ee460bc8c3d9e44b94f5710c5c6f942 not found: ID does not exist" Jul 07 02:35:12.356903 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.356543 2564 scope.go:117] "RemoveContainer" containerID="5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9" Jul 07 02:35:12.357083 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:35:12.357039 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9\": container with ID starting with 5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9 not found: ID does not exist" containerID="5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9" Jul 07 02:35:12.357216 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.357081 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9"} err="failed to get container status \"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9\": rpc error: code = NotFound desc = could not find container \"5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9\": container with ID starting with 5eccb35e3b81c80a7c3c4a65b138ba63284f1bf0062e386e1fa071beeeb2c0d9 not found: ID does not exist" Jul 07 02:35:12.359596 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.359558 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:35:12.364059 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.364018 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-m74kj"] Jul 07 02:35:12.425538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425490 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301c6a18-a6ef-440d-a19f-764402eb980c-istio-podinfo\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425531 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-envoy\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425549 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-workload-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425560 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301c6a18-a6ef-440d-a19f-764402eb980c-istiod-ca-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425573 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgprz\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-kube-api-access-xgprz\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425584 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301c6a18-a6ef-440d-a19f-764402eb980c-istio-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425596 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-credential-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:12.425843 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:12.425611 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301c6a18-a6ef-440d-a19f-764402eb980c-istio-data\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:13.791606 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:13.791564 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" path="/var/lib/kubelet/pods/301c6a18-a6ef-440d-a19f-764402eb980c/volumes" Jul 07 02:35:18.093202 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:18.093165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:19.942904 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:19.942865 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:19.962106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:19.962044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:24.979143 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:24.979107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:25.278692 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:25.278593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:30.970118 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:30.970074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:34.965064 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:34.964997 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" podUID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" containerName="main" containerID="cri-o://87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336" gracePeriod=45 Jul 07 02:35:35.238758 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.238719 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:35.337608 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337567 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rh8n\" (UniqueName: \"kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.337847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.337847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337665 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.337847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.337847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337755 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.337847 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.337815 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir\") pod \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\" (UID: \"f87801fb-0437-4fe2-a120-94a3f2a4bc6d\") " Jul 07 02:35:35.338130 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.338082 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:35.338191 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.338120 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:35.338191 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.338067 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home" (OuterVolumeSpecName: "home") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:35.340242 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.340198 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm" (OuterVolumeSpecName: "dshm") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:35:35.340386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.340244 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:35:35.340386 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.340281 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n" (OuterVolumeSpecName: "kube-api-access-6rh8n") pod "f87801fb-0437-4fe2-a120-94a3f2a4bc6d" (UID: "f87801fb-0437-4fe2-a120-94a3f2a4bc6d"). InnerVolumeSpecName "kube-api-access-6rh8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:35:35.438721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438658 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.438721 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438713 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.438995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438752 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.438995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438764 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rh8n\" (UniqueName: \"kubernetes.io/projected/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-kube-api-access-6rh8n\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.438995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438777 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.438995 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.438824 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f87801fb-0437-4fe2-a120-94a3f2a4bc6d-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:35:35.454889 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.454845 2564 generic.go:356] "Generic (PLEG): container finished" podID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" containerID="87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336" exitCode=0 Jul 07 02:35:35.455111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.454909 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" Jul 07 02:35:35.455111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.454951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" event={"ID":"f87801fb-0437-4fe2-a120-94a3f2a4bc6d","Type":"ContainerDied","Data":"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336"} Jul 07 02:35:35.455111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.455027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl" event={"ID":"f87801fb-0437-4fe2-a120-94a3f2a4bc6d","Type":"ContainerDied","Data":"fe8033de3572e0fd0db15a0b0c82fe86b550cb013f013efbfe5aab4fa4003f43"} Jul 07 02:35:35.455111 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.455060 2564 scope.go:117] "RemoveContainer" containerID="87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336" Jul 07 02:35:35.465271 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.465252 2564 scope.go:117] "RemoveContainer" containerID="87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336" Jul 07 02:35:35.465637 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:35:35.465615 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336\": container with ID starting with 87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336 not found: ID does not exist" containerID="87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336" Jul 07 02:35:35.465701 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.465653 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336"} err="failed to get container status \"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336\": rpc error: code = NotFound desc = could not find container \"87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336\": container with ID starting with 87916072fb2c79715bf3f942f6f90a162937a1118b7d17b3b351636997e52336 not found: ID does not exist" Jul 07 02:35:35.477906 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.477806 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:35.483805 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.483754 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bp4xnl"] Jul 07 02:35:35.796899 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:35.796847 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" path="/var/lib/kubelet/pods/f87801fb-0437-4fe2-a120-94a3f2a4bc6d/volumes" Jul 07 02:35:50.221277 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:50.221193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:35:50.283274 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:35:50.283239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:21.147990 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.147949 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148301 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148315 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148329 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" containerName="main" Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148338 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" containerName="main" Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148412 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87801fb-0437-4fe2-a120-94a3f2a4bc6d" containerName="main" Jul 07 02:36:21.148479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.148421 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:36:21.151428 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.151398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.155257 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.155223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:36:21.155257 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.155254 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 02:36:21.155494 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.155306 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\"" Jul 07 02:36:21.155494 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.155372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:36:21.160361 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.160334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:36:21.164606 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.164576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:21.249243 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.249465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksm76\" (UniqueName: \"kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.249465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.249465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.249465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.249465 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.249436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350473 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksm76\" (UniqueName: \"kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.350679 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.350609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.351035 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.351013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.351075 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.351051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.351140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.351078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.353229 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.353204 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.353463 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.353444 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.359777 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.359729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksm76\" (UniqueName: \"kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.464733 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.464612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:21.501329 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.501274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:21.621164 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.621132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:36:21.623957 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:36:21.623914 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf016f4_d67a_4e31_9cb0_deb8af2d478d.slice/crio-bf08175598b9e0866c0eb8bc6ac9d2a5380ce2ac11879b373c49c2b789b4e19e WatchSource:0}: Error finding container bf08175598b9e0866c0eb8bc6ac9d2a5380ce2ac11879b373c49c2b789b4e19e: Status 404 returned error can't find the container with id bf08175598b9e0866c0eb8bc6ac9d2a5380ce2ac11879b373c49c2b789b4e19e Jul 07 02:36:21.698561 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.698497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" event={"ID":"5bf016f4-d67a-4e31-9cb0-deb8af2d478d","Type":"ContainerStarted","Data":"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c"} Jul 07 02:36:21.698561 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.698561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" event={"ID":"5bf016f4-d67a-4e31-9cb0-deb8af2d478d","Type":"ContainerStarted","Data":"bf08175598b9e0866c0eb8bc6ac9d2a5380ce2ac11879b373c49c2b789b4e19e"} Jul 07 02:36:21.717619 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:21.717471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" podStartSLOduration=0.717450728 podStartE2EDuration="717.450728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:36:21.717242393 +0000 UTC m=+1106.474791455" watchObservedRunningTime="2026-07-07 02:36:21.717450728 +0000 UTC m=+1106.474999753" Jul 07 02:36:22.464888 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:22.464850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:31.465636 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:31.465584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:31.479011 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:31.478979 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:31.764346 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:31.764256 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:36:46.027378 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:46.027328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:55.566250 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:55.566207 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:36:55.571766 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:55.571725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:36:55.658677 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:36:55.658643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:10.591858 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.591810 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerName="main" containerID="cri-o://fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c" gracePeriod=45 Jul 07 02:37:10.753882 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.753832 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused" Jul 07 02:37:10.854737 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.854658 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:37:10.888975 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.888936 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.888975 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.888981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksm76\" (UniqueName: \"kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.889247 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889015 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.889247 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889077 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.890741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889506 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:37:10.890741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.890741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home\") pod \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\" (UID: \"5bf016f4-d67a-4e31-9cb0-deb8af2d478d\") " Jul 07 02:37:10.890741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.889738 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:37:10.890741 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.890086 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home" (OuterVolumeSpecName: "home") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:37:10.897664 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.892725 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.897664 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.892756 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.897664 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.892771 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.898102 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.897971 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm" (OuterVolumeSpecName: "dshm") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:37:10.898102 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.897994 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:37:10.900405 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.900372 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76" (OuterVolumeSpecName: "kube-api-access-ksm76") pod "5bf016f4-d67a-4e31-9cb0-deb8af2d478d" (UID: "5bf016f4-d67a-4e31-9cb0-deb8af2d478d"). InnerVolumeSpecName "kube-api-access-ksm76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:37:10.968341 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.968301 2564 generic.go:356] "Generic (PLEG): container finished" podID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerID="fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c" exitCode=0 Jul 07 02:37:10.968525 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.968362 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" Jul 07 02:37:10.968525 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.968360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" event={"ID":"5bf016f4-d67a-4e31-9cb0-deb8af2d478d","Type":"ContainerDied","Data":"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c"} Jul 07 02:37:10.968525 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.968446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb" event={"ID":"5bf016f4-d67a-4e31-9cb0-deb8af2d478d","Type":"ContainerDied","Data":"bf08175598b9e0866c0eb8bc6ac9d2a5380ce2ac11879b373c49c2b789b4e19e"} Jul 07 02:37:10.968525 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.968471 2564 scope.go:117] "RemoveContainer" containerID="fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c" Jul 07 02:37:10.978106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.978085 2564 scope.go:117] "RemoveContainer" containerID="fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c" Jul 07 02:37:10.978410 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:37:10.978391 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c\": container with ID starting with fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c not found: ID does not exist" containerID="fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c" Jul 07 02:37:10.978488 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.978417 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c"} err="failed to get container status \"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c\": rpc error: code = NotFound desc = could not find container \"fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c\": container with ID starting with fb886f1ab9f352d0b378661dd01a256f69de5b92a06784bf7f595dbe159d2a3c not found: ID does not exist" Jul 07 02:37:10.993624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.993586 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.993624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.993618 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.993624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.993629 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksm76\" (UniqueName: \"kubernetes.io/projected/5bf016f4-d67a-4e31-9cb0-deb8af2d478d-kube-api-access-ksm76\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:37:10.994616 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.994593 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:37:10.999065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:10.999026 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879rt4nb"] Jul 07 02:37:11.791653 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:11.791609 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" path="/var/lib/kubelet/pods/5bf016f4-d67a-4e31-9cb0-deb8af2d478d/volumes" Jul 07 02:37:26.050688 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.050602 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051009 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerName="main" Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051022 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerName="main" Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051030 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051035 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051084 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="301c6a18-a6ef-440d-a19f-764402eb980c" containerName="istio-proxy" Jul 07 02:37:26.051112 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.051093 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf016f4-d67a-4e31-9cb0-deb8af2d478d" containerName="main" Jul 07 02:37:26.053162 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.053140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.057378 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.057347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 02:37:26.057646 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.057626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:37:26.057803 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.057699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\"" Jul 07 02:37:26.057803 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.057724 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:37:26.062848 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.062815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:37:26.131704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhdq\" (UniqueName: \"kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.131704 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.132009 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.132009 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.132009 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.132009 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.131975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.141210 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.141174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:26.232964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.232926 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.232964 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.232972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233245 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.232996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233245 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233245 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhdq\" (UniqueName: \"kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233245 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233469 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233435 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233469 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.233570 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.233522 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.235753 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.235726 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.235957 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.235937 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.242221 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.242188 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhdq\" (UniqueName: \"kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.365624 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.365521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:26.480616 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.480575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:26.524227 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:26.524130 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:37:26.525774 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:37:26.525740 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabebaf9_b43a_468c_87c3_9309240fa8de.slice/crio-a5261ee9648da165eb3967e44ba2db034857480afee04d92c441cb439f708f61 WatchSource:0}: Error finding container a5261ee9648da165eb3967e44ba2db034857480afee04d92c441cb439f708f61: Status 404 returned error can't find the container with id a5261ee9648da165eb3967e44ba2db034857480afee04d92c441cb439f708f61 Jul 07 02:37:27.052097 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:27.052056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" event={"ID":"fabebaf9-b43a-468c-87c3-9309240fa8de","Type":"ContainerStarted","Data":"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7"} Jul 07 02:37:27.052518 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:27.052135 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" event={"ID":"fabebaf9-b43a-468c-87c3-9309240fa8de","Type":"ContainerStarted","Data":"a5261ee9648da165eb3967e44ba2db034857480afee04d92c441cb439f708f61"} Jul 07 02:37:27.071245 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:27.071181 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" podStartSLOduration=1.071160287 podStartE2EDuration="1.071160287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:37:27.069429696 +0000 UTC m=+1171.826978733" watchObservedRunningTime="2026-07-07 02:37:27.071160287 +0000 UTC m=+1171.828709352" Jul 07 02:37:27.366140 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:27.366027 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:36.366260 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:36.366209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:36.379460 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:36.379426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:37.117283 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:37.117252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:37:46.840161 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:46.840101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:48.467426 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:48.467392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:37:55.745021 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:55.744974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:37:55.747369 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:37:55.747338 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:38:09.703736 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:09.703684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:09.764934 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:09.764897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.375074 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:33.375036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:33.611441 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:33.611407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:51.767203 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:51.767111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:52.012447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:52.012399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:38:58.872990 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:38:58.872956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:22.961753 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:22.961710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:27.694246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:27.694200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:27.789805 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:27.789744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:47.491346 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:47.491299 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:39:47.493144 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:47.493076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:39:47.560129 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:39:47.560085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:02.515186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.515140 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" podUID="fabebaf9-b43a-468c-87c3-9309240fa8de" containerName="main" containerID="cri-o://29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7" gracePeriod=45 Jul 07 02:40:02.764851 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.764813 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:40:02.885701 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.885661 2564 generic.go:356] "Generic (PLEG): container finished" podID="fabebaf9-b43a-468c-87c3-9309240fa8de" containerID="29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7" exitCode=0 Jul 07 02:40:02.885928 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.885711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" Jul 07 02:40:02.885928 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.885752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" event={"ID":"fabebaf9-b43a-468c-87c3-9309240fa8de","Type":"ContainerDied","Data":"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7"} Jul 07 02:40:02.885928 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.885845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7" event={"ID":"fabebaf9-b43a-468c-87c3-9309240fa8de","Type":"ContainerDied","Data":"a5261ee9648da165eb3967e44ba2db034857480afee04d92c441cb439f708f61"} Jul 07 02:40:02.885928 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.885872 2564 scope.go:117] "RemoveContainer" containerID="29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7" Jul 07 02:40:02.895883 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.895860 2564 scope.go:117] "RemoveContainer" containerID="29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7" Jul 07 02:40:02.896276 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:40:02.896254 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7\": container with ID starting with 29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7 not found: ID does not exist" containerID="29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7" Jul 07 02:40:02.896332 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.896290 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7"} err="failed to get container status \"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7\": rpc error: code = NotFound desc = could not find container \"29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7\": container with ID starting with 29c545ee3797765f16a26ff298aec32eee239ba4d29666558bb119d094c364d7 not found: ID does not exist" Jul 07 02:40:02.949328 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949282 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzhdq\" (UniqueName: \"kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949380 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949538 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949756 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir\") pod \"fabebaf9-b43a-468c-87c3-9309240fa8de\" (UID: \"fabebaf9-b43a-468c-87c3-9309240fa8de\") " Jul 07 02:40:02.949756 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949610 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home" (OuterVolumeSpecName: "home") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:40:02.949904 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949761 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache" (OuterVolumeSpecName: "model-cache") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:40:02.949904 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949825 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:40:02.949904 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.949840 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:02.951939 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.951906 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq" (OuterVolumeSpecName: "kube-api-access-bzhdq") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "kube-api-access-bzhdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:40:02.951939 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.951908 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:40:02.952085 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:02.951936 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm" (OuterVolumeSpecName: "dshm") pod "fabebaf9-b43a-468c-87c3-9309240fa8de" (UID: "fabebaf9-b43a-468c-87c3-9309240fa8de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:40:03.051231 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.051188 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:03.051231 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.051220 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhdq\" (UniqueName: \"kubernetes.io/projected/fabebaf9-b43a-468c-87c3-9309240fa8de-kube-api-access-bzhdq\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:03.051231 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.051231 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:03.051231 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.051240 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabebaf9-b43a-468c-87c3-9309240fa8de-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:03.051231 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.051249 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabebaf9-b43a-468c-87c3-9309240fa8de-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:40:03.212073 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.212025 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:40:03.214070 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.214042 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4s4vv7"] Jul 07 02:40:03.791653 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:03.791611 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fabebaf9-b43a-468c-87c3-9309240fa8de" path="/var/lib/kubelet/pods/fabebaf9-b43a-468c-87c3-9309240fa8de/volumes" Jul 07 02:40:22.242740 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.242645 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:40:22.243239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.243035 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabebaf9-b43a-468c-87c3-9309240fa8de" containerName="main" Jul 07 02:40:22.243239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.243048 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabebaf9-b43a-468c-87c3-9309240fa8de" containerName="main" Jul 07 02:40:22.243239 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.243131 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fabebaf9-b43a-468c-87c3-9309240fa8de" containerName="main" Jul 07 02:40:22.246199 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.246175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.250819 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.250774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\"" Jul 07 02:40:22.251016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.250822 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:40:22.251016 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.250822 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 02:40:22.251584 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.251563 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:40:22.257464 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.257424 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:40:22.271974 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.271942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:22.312286 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptj9\" (UniqueName: \"kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.312503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.312503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.312503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.312503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.312503 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.312452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413660 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptj9\" (UniqueName: \"kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.413881 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.413855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.414215 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.414194 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.414281 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.414260 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.414380 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.414355 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.416129 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.416094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.416440 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.416422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.423434 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.423403 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptj9\" (UniqueName: \"kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.516040 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.515929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:22.557673 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.557626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:22.707513 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.707447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:40:22.710175 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:40:22.710144 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c662b96_924d_4753_a066_c15d03c63e46.slice/crio-1111c4a29f50dad58e9f807c48ba9b47c915502dfbc4c1bf8d59234129e6083c WatchSource:0}: Error finding container 1111c4a29f50dad58e9f807c48ba9b47c915502dfbc4c1bf8d59234129e6083c: Status 404 returned error can't find the container with id 1111c4a29f50dad58e9f807c48ba9b47c915502dfbc4c1bf8d59234129e6083c Jul 07 02:40:22.992214 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.992175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" event={"ID":"0c662b96-924d-4753-a066-c15d03c63e46","Type":"ContainerStarted","Data":"10d3a6247c2dd126e80e2c838d82dc5b029c925068228fc3207e1d48b5961f27"} Jul 07 02:40:22.992428 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:22.992241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" event={"ID":"0c662b96-924d-4753-a066-c15d03c63e46","Type":"ContainerStarted","Data":"1111c4a29f50dad58e9f807c48ba9b47c915502dfbc4c1bf8d59234129e6083c"} Jul 07 02:40:23.012476 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:23.012424 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" podStartSLOduration=1.012408184 podStartE2EDuration="1.012408184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:40:23.011964109 +0000 UTC m=+1347.769513161" watchObservedRunningTime="2026-07-07 02:40:23.012408184 +0000 UTC m=+1347.769957250" Jul 07 02:40:23.558183 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:23.558130 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:30.145989 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:30.145951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:31.893642 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:31.893610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:32.558578 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:32.558533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:32.571303 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:32.571270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:33.057603 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:33.057564 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:40:53.476199 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:53.476160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:40:53.534585 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:40:53.534547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:02.329953 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:02.329914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:02.797006 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:02.796964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:16.845601 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:16.845558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:29.499863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:29.499825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:29.824283 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:29.824247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:40.923982 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:40.923941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:45.071024 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:45.070966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:45.142320 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:45.142279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:41:45.568305 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:41:45.568268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:05.367533 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:05.367492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:42:06.165875 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:06.165826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:06.266697 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:06.266651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:09.838996 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:09.838953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:20.393022 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.392972 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" podUID="0c662b96-924d-4753-a066-c15d03c63e46" containerName="main" containerID="cri-o://10d3a6247c2dd126e80e2c838d82dc5b029c925068228fc3207e1d48b5961f27" gracePeriod=45 Jul 07 02:42:20.623528 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.623478 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c662b96-924d-4753-a066-c15d03c63e46" containerID="10d3a6247c2dd126e80e2c838d82dc5b029c925068228fc3207e1d48b5961f27" exitCode=0 Jul 07 02:42:20.623726 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.623553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" event={"ID":"0c662b96-924d-4753-a066-c15d03c63e46","Type":"ContainerDied","Data":"10d3a6247c2dd126e80e2c838d82dc5b029c925068228fc3207e1d48b5961f27"} Jul 07 02:42:20.662993 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.662959 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:42:20.770923 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.770877 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.770965 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.770990 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fptj9\" (UniqueName: \"kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771131 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache\") pod \"0c662b96-924d-4753-a066-c15d03c63e46\" (UID: \"0c662b96-924d-4753-a066-c15d03c63e46\") " Jul 07 02:42:20.771397 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771245 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home" (OuterVolumeSpecName: "home") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:42:20.771397 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771347 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:42:20.771478 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.771416 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:42:20.773489 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.773443 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:42:20.773489 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.773448 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9" (OuterVolumeSpecName: "kube-api-access-fptj9") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "kube-api-access-fptj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:42:20.773723 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.773605 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm" (OuterVolumeSpecName: "dshm") pod "0c662b96-924d-4753-a066-c15d03c63e46" (UID: "0c662b96-924d-4753-a066-c15d03c63e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:42:20.873054 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873012 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:20.873054 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873054 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:20.873054 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873077 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fptj9\" (UniqueName: \"kubernetes.io/projected/0c662b96-924d-4753-a066-c15d03c63e46-kube-api-access-fptj9\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:20.873342 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873093 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:20.873342 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873111 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c662b96-924d-4753-a066-c15d03c63e46-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:20.873342 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:20.873123 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c662b96-924d-4753-a066-c15d03c63e46-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:42:21.630065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.630014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" event={"ID":"0c662b96-924d-4753-a066-c15d03c63e46","Type":"ContainerDied","Data":"1111c4a29f50dad58e9f807c48ba9b47c915502dfbc4c1bf8d59234129e6083c"} Jul 07 02:42:21.630065 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.630044 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz" Jul 07 02:42:21.630582 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.630099 2564 scope.go:117] "RemoveContainer" containerID="10d3a6247c2dd126e80e2c838d82dc5b029c925068228fc3207e1d48b5961f27" Jul 07 02:42:21.654628 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.654585 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:42:21.658934 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.658886 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-hrrbz"] Jul 07 02:42:21.791510 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:21.791467 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c662b96-924d-4753-a066-c15d03c63e46" path="/var/lib/kubelet/pods/0c662b96-924d-4753-a066-c15d03c63e46/volumes" Jul 07 02:42:32.965408 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:32.965369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:33.293730 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:33.293696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:41.689734 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:41.689683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:42:55.778240 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:55.778206 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:42:55.782914 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:42:55.782888 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:43:45.373730 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:43:45.373622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:43:45.474906 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:43:45.474866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:05.793909 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.793865 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:05.794533 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.794326 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c662b96-924d-4753-a066-c15d03c63e46" containerName="main" Jul 07 02:44:05.794533 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.794346 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c662b96-924d-4753-a066-c15d03c63e46" containerName="main" Jul 07 02:44:05.794533 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.794441 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c662b96-924d-4753-a066-c15d03c63e46" containerName="main" Jul 07 02:44:05.797950 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.797915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.801351 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.801318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\"" Jul 07 02:44:05.801593 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.801574 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" Jul 07 02:44:05.801831 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.801813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:44:05.802051 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.802023 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:44:05.803736 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.803706 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:05.908380 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908380 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908467 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908681 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bx9\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:05.908910 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:05.908749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009295 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bx9\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.009655 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.010078 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009901 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.010078 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.009925 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.010236 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.010096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.010236 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.010160 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.010585 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.010561 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.012155 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.012129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.012379 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.012360 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.018288 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.018245 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.018467 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.018314 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bx9\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9\") pod \"router-gateway-1-openshift-default-799f46c59b-qt5dn\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.111895 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.111770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:06.282293 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.282259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:06.285410 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:44:06.285368 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9469ef7_591c_4075_b139_5e91d327fbf5.slice/crio-d1484240ed2bd055c27d21ce7316e95eb9fc27ac030be07ebbc53a959baa890f WatchSource:0}: Error finding container d1484240ed2bd055c27d21ce7316e95eb9fc27ac030be07ebbc53a959baa890f: Status 404 returned error can't find the container with id d1484240ed2bd055c27d21ce7316e95eb9fc27ac030be07ebbc53a959baa890f Jul 07 02:44:06.288142 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.288097 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:44:06.288272 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.288197 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:44:06.288272 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.288246 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 02:44:06.633758 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:06.633714 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:07.228082 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:07.228025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" event={"ID":"f9469ef7-591c-4075-b139-5e91d327fbf5","Type":"ContainerStarted","Data":"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e"} Jul 07 02:44:07.228510 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:07.228108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" event={"ID":"f9469ef7-591c-4075-b139-5e91d327fbf5","Type":"ContainerStarted","Data":"d1484240ed2bd055c27d21ce7316e95eb9fc27ac030be07ebbc53a959baa890f"} Jul 07 02:44:07.251222 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:07.251168 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" podStartSLOduration=2.2511511779999998 podStartE2EDuration="2.251151178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:44:07.248568867 +0000 UTC m=+1572.006117909" watchObservedRunningTime="2026-07-07 02:44:07.251151178 +0000 UTC m=+1572.008700259" Jul 07 02:44:08.112837 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:08.112798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:08.118607 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:08.118562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:08.233001 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:08.232957 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" podUID="f9469ef7-591c-4075-b139-5e91d327fbf5" containerName="istio-proxy" containerID="cri-o://cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e" gracePeriod=30 Jul 07 02:44:08.233495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:08.233077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:08.234241 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:08.234221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:13.506644 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.506560 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:13.681476 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681411 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681476 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681476 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681562 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.681897 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.681704 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9bx9\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9\") pod \"f9469ef7-591c-4075-b139-5e91d327fbf5\" (UID: \"f9469ef7-591c-4075-b139-5e91d327fbf5\") " Jul 07 02:44:13.682248 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.682182 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:44:13.682382 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.682333 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:44:13.682492 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.682412 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:44:13.682565 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.682494 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:44:13.682760 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.682725 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 02:44:13.684186 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.684158 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:44:13.684493 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.684474 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 02:44:13.684757 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.684736 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9" (OuterVolumeSpecName: "kube-api-access-g9bx9") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "kube-api-access-g9bx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:44:13.684887 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.684861 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9469ef7-591c-4075-b139-5e91d327fbf5" (UID: "f9469ef7-591c-4075-b139-5e91d327fbf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783582 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9469ef7-591c-4075-b139-5e91d327fbf5-istiod-ca-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783625 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-credential-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783641 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bx9\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-kube-api-access-g9bx9\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783655 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-podinfo\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783669 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-envoy\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783683 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783696 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-workload-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783709 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-data\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:13.783810 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:13.783730 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9469ef7-591c-4075-b139-5e91d327fbf5-istio-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:44:14.270907 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.270763 2564 generic.go:356] "Generic (PLEG): container finished" podID="f9469ef7-591c-4075-b139-5e91d327fbf5" containerID="cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e" exitCode=0 Jul 07 02:44:14.270907 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.270837 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" Jul 07 02:44:14.270907 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.270854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" event={"ID":"f9469ef7-591c-4075-b139-5e91d327fbf5","Type":"ContainerDied","Data":"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e"} Jul 07 02:44:14.271201 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.270963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn" event={"ID":"f9469ef7-591c-4075-b139-5e91d327fbf5","Type":"ContainerDied","Data":"d1484240ed2bd055c27d21ce7316e95eb9fc27ac030be07ebbc53a959baa890f"} Jul 07 02:44:14.271201 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.271001 2564 scope.go:117] "RemoveContainer" containerID="cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e" Jul 07 02:44:14.281209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.281181 2564 scope.go:117] "RemoveContainer" containerID="cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e" Jul 07 02:44:14.281629 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:44:14.281583 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e\": container with ID starting with cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e not found: ID does not exist" containerID="cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e" Jul 07 02:44:14.281629 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.281617 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e"} err="failed to get container status \"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e\": rpc error: code = NotFound desc = could not find container \"cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e\": container with ID starting with cdc0d01052594d875d67755a84dcc5d62d0996175005db89de92cd140e76118e not found: ID does not exist" Jul 07 02:44:14.291546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.291499 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:14.295427 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:14.295394 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-qt5dn"] Jul 07 02:44:15.794715 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:15.794669 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9469ef7-591c-4075-b139-5e91d327fbf5" path="/var/lib/kubelet/pods/f9469ef7-591c-4075-b139-5e91d327fbf5/volumes" Jul 07 02:44:19.056560 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.056503 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:44:19.057116 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.057076 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9469ef7-591c-4075-b139-5e91d327fbf5" containerName="istio-proxy" Jul 07 02:44:19.057116 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.057102 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9469ef7-591c-4075-b139-5e91d327fbf5" containerName="istio-proxy" Jul 07 02:44:19.057214 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.057192 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9469ef7-591c-4075-b139-5e91d327fbf5" containerName="istio-proxy" Jul 07 02:44:19.062334 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.062298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.065620 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.065588 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 02:44:19.065842 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.065588 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 02:44:19.067132 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.067089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 02:44:19.067318 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.067196 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\"" Jul 07 02:44:19.070205 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.070172 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:44:19.131305 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.131305 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.131551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.131551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lct9d\" (UniqueName: \"kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.131551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.131551 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.131508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.200183 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.200143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:19.232408 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.232631 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.232631 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.232631 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.232631 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.232631 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.232575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lct9d\" (UniqueName: \"kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.233044 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.233018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.233106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.233048 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.233106 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.233059 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.235301 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.235267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.235539 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.235520 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.244400 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.244368 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lct9d\" (UniqueName: \"kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.375848 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.375704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:19.549365 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:19.547738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:44:19.553654 ip-10-0-137-109 kubenswrapper[2564]: W0707 02:44:19.553615 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36b017c_3a7b_4bc4_80af_dd5b3d3322c8.slice/crio-fd8ca0097b8a4814abcaca86441cc972bf3067a6cc03512cbc4aa3af107963d5 WatchSource:0}: Error finding container fd8ca0097b8a4814abcaca86441cc972bf3067a6cc03512cbc4aa3af107963d5: Status 404 returned error can't find the container with id fd8ca0097b8a4814abcaca86441cc972bf3067a6cc03512cbc4aa3af107963d5 Jul 07 02:44:20.307495 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:20.307453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" event={"ID":"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8","Type":"ContainerStarted","Data":"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e"} Jul 07 02:44:20.308032 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:20.307510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" event={"ID":"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8","Type":"ContainerStarted","Data":"fd8ca0097b8a4814abcaca86441cc972bf3067a6cc03512cbc4aa3af107963d5"} Jul 07 02:44:20.325972 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:20.325902 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" podStartSLOduration=1.325880273 podStartE2EDuration="1.325880273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:44:20.324732515 +0000 UTC m=+1585.082281560" watchObservedRunningTime="2026-07-07 02:44:20.325880273 +0000 UTC m=+1585.083429298" Jul 07 02:44:20.376209 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:20.376169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:27.579390 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:27.579349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:28.784450 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:28.784396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:29.375969 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:29.375928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:29.388767 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:29.388734 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:29.617204 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:29.617170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:29.681654 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:29.681558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:30.383723 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:30.383694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:44:53.949480 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:53.949391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:44:54.211233 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:44:54.211136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:45:04.450835 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:45:04.450763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:15.989644 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:15.989558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:16.048008 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:16.047976 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:16.433311 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:16.433280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:16.491447 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:16.491406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:36.422566 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:36.422533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:36.524530 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:36.524496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:36.626048 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:36.626018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:39.056300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:39.056264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:39.157927 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:39.157885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:46:39.569049 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:46:39.569008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:02.083508 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:02.083464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:02.498925 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:02.498749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:05.737456 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:05.737414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:12.938510 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:12.938465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:47:55.812890 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:55.812861 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:47:55.818347 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:47:55.818320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:48:22.947058 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:48:22.947011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:09.576724 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:49:09.576679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:09.652543 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:49:09.652485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:33.540040 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:49:33.539942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:34.664937 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:49:34.664905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:49:40.539852 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:49:40.539817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:51:54.478479 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:51:54.478441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:27.643414 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:27.643321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:27.743483 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:27.743446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:55.261815 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.261760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:55.380199 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.380166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:55.538246 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.538213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:52:55.849208 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.849116 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:52:55.855391 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.855361 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:52:55.948556 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:52:55.948525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:53:01.818537 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:53:01.818499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:55:35.884979 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:35.884939 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:55:35.939062 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:35.939022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:55:50.907075 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:50.907015 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" podUID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" containerName="main" containerID="cri-o://7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e" gracePeriod=45 Jul 07 02:55:51.158297 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.158210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:55:51.235300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235266 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235300 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235309 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235554 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lct9d\" (UniqueName: \"kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235554 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235393 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235554 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235554 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235466 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir\") pod \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\" (UID: \"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8\") " Jul 07 02:55:51.235707 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235608 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:55:51.235772 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235749 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home" (OuterVolumeSpecName: "home") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:55:51.235885 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.235856 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:55:51.237804 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.237757 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm" (OuterVolumeSpecName: "dshm") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 02:55:51.237902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.237771 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 02:55:51.237902 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.237859 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d" (OuterVolumeSpecName: "kube-api-access-lct9d") pod "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" (UID: "a36b017c-3a7b-4bc4-80af-dd5b3d3322c8"). InnerVolumeSpecName "kube-api-access-lct9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336515 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336542 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336553 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336560 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336569 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lct9d\" (UniqueName: \"kubernetes.io/projected/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-kube-api-access-lct9d\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.336548 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.336577 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 02:55:51.947487 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.947447 2564 generic.go:356] "Generic (PLEG): container finished" podID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" containerID="7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e" exitCode=0 Jul 07 02:55:51.947967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.947502 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" Jul 07 02:55:51.947967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.947517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" event={"ID":"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8","Type":"ContainerDied","Data":"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e"} Jul 07 02:55:51.947967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.947570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h" event={"ID":"a36b017c-3a7b-4bc4-80af-dd5b3d3322c8","Type":"ContainerDied","Data":"fd8ca0097b8a4814abcaca86441cc972bf3067a6cc03512cbc4aa3af107963d5"} Jul 07 02:55:51.947967 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.947601 2564 scope.go:117] "RemoveContainer" containerID="7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e" Jul 07 02:55:51.957047 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.957016 2564 scope.go:117] "RemoveContainer" containerID="7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e" Jul 07 02:55:51.957379 ip-10-0-137-109 kubenswrapper[2564]: E0707 02:55:51.957353 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e\": container with ID starting with 7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e not found: ID does not exist" containerID="7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e" Jul 07 02:55:51.957452 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.957391 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e"} err="failed to get container status \"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e\": rpc error: code = NotFound desc = could not find container \"7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e\": container with ID starting with 7ae929b53fd38d6e051a91bb23f0e864cc1b385664177c7d2dc1b5cede7ae98e not found: ID does not exist" Jul 07 02:55:51.969408 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.969368 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:55:51.974710 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:51.974677 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4dtv7h"] Jul 07 02:55:53.792169 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:55:53.792120 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" path="/var/lib/kubelet/pods/a36b017c-3a7b-4bc4-80af-dd5b3d3322c8/volumes" Jul 07 02:56:00.794319 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:00.794282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:01.060567 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:01.060463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:06.444446 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:06.444402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:15.071863 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:15.071827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:16.399700 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:16.399665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:18.144959 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:18.144919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:18.243189 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:18.243149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:59.157609 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:59.157522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:56:59.685137 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:56:59.685101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:57:07.723056 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:57:07.723018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:57:51.675518 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:57:51.675480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:57:55.883420 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:57:55.883383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:57:55.891098 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:57:55.891067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 02:58:00.124859 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:00.124821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:00.225474 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:00.225434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:05.686966 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:05.686920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:05.746717 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:05.746680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:32.058729 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:32.058649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:32.384976 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:32.384854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:58:37.135511 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:58:37.135474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:00.739406 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:00.739373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:00.924299 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:00.924259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:01.197845 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:01.197756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:18.596332 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:18.596294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:18.843887 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:18.843854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 02:59:24.987546 ip-10-0-137-109 kubenswrapper[2564]: I0707 02:59:24.987512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:00:39.346106 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:39.346055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:00:40.860890 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:40.860843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:00:59.792098 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.792066 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:00:59.792471 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.792365 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" containerName="main" Jul 07 03:00:59.792471 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.792376 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" containerName="main" Jul 07 03:00:59.792471 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.792439 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36b017c-3a7b-4bc4-80af-dd5b3d3322c8" containerName="main" Jul 07 03:00:59.795245 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.795226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.798300 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.798278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" Jul 07 03:00:59.798428 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.798336 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 03:00:59.798742 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.798719 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 03:00:59.798902 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.798870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\"" Jul 07 03:00:59.799696 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.799676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:00:59.930316 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930316 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930537 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930537 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbq7\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930537 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930537 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930537 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930697 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:00:59.930697 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:00:59.930603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031121 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031121 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031376 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031376 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031376 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031536 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031536 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbq7\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031536 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031536 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031732 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031541 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031732 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031835 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.031875 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.031849 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.032143 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.032124 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.033816 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.033763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.033868 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.033772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.041058 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.041026 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.041165 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.041150 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbq7\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7\") pod \"router-gateway-1-openshift-default-799f46c59b-gb7nq\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.108631 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.108547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:00.248333 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.248307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:01:00.250329 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:01:00.250301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb755f034_0fb2_4303_a629_2cefd1c08947.slice/crio-b3da8df9acb98025a2a39480aea9bbfa37219f4f65bd8f9e1b7d177019ba24aa WatchSource:0}: Error finding container b3da8df9acb98025a2a39480aea9bbfa37219f4f65bd8f9e1b7d177019ba24aa: Status 404 returned error can't find the container with id b3da8df9acb98025a2a39480aea9bbfa37219f4f65bd8f9e1b7d177019ba24aa Jul 07 03:01:00.252247 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.252212 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:00.252344 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.252278 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:00.252344 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.252309 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:00.507140 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.507045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerStarted","Data":"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5"} Jul 07 03:01:00.507140 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.507102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerStarted","Data":"b3da8df9acb98025a2a39480aea9bbfa37219f4f65bd8f9e1b7d177019ba24aa"} Jul 07 03:01:00.529849 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:00.529799 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podStartSLOduration=1.5297667750000001 podStartE2EDuration="1.529766775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:01:00.52884168 +0000 UTC m=+2585.286390727" watchObservedRunningTime="2026-07-07 03:01:00.529766775 +0000 UTC m=+2585.287315800" Jul 07 03:01:01.108770 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.108738 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:01.113858 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.113828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:01.512503 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.512421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:01.513199 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.513187 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:01.769274 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.769195 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s"] Jul 07 03:01:01.772308 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.772288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.785168 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.785141 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s"] Jul 07 03:01:01.847902 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.847875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.847909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2d15b-598d-452b-8b55-38ea546e380d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.847939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s977g\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-kube-api-access-s977g\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.847961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.847979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.848000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.848014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848069 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.848036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.848299 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.848105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.948950 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.948914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.948950 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.948951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2d15b-598d-452b-8b55-38ea546e380d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.948989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s977g\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-kube-api-access-s977g\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949192 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949543 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949543 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949654 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949603 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949709 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949668 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.949854 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949831 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd2d15b-598d-452b-8b55-38ea546e380d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.950014 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.949938 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.951753 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.951729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.951864 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.951773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.957709 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.957676 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s977g\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-kube-api-access-s977g\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:01.957824 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:01.957683 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd2d15b-598d-452b-8b55-38ea546e380d-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-tzh7s\" (UID: \"7bd2d15b-598d-452b-8b55-38ea546e380d\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:02.084452 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.084422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:02.231003 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.230979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s"] Jul 07 03:01:02.232858 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:01:02.232828 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd2d15b_598d_452b_8b55_38ea546e380d.slice/crio-7a0bbec6f8f5dab03af8b2030e1684b32fd0b4ed12cb0124c6808727ae566e31 WatchSource:0}: Error finding container 7a0bbec6f8f5dab03af8b2030e1684b32fd0b4ed12cb0124c6808727ae566e31: Status 404 returned error can't find the container with id 7a0bbec6f8f5dab03af8b2030e1684b32fd0b4ed12cb0124c6808727ae566e31 Jul 07 03:01:02.235071 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.235032 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:02.235149 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.235109 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:02.235149 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.235141 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:02.518701 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.518609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" event={"ID":"7bd2d15b-598d-452b-8b55-38ea546e380d","Type":"ContainerStarted","Data":"769830b156d946e94b4c6a3235666aaf988c4a87aa931bfc2f1812531b4a8917"} Jul 07 03:01:02.518890 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.518717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" event={"ID":"7bd2d15b-598d-452b-8b55-38ea546e380d","Type":"ContainerStarted","Data":"7a0bbec6f8f5dab03af8b2030e1684b32fd0b4ed12cb0124c6808727ae566e31"} Jul 07 03:01:02.547136 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:02.547085 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podStartSLOduration=1.547070038 podStartE2EDuration="1.547070038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:01:02.546570212 +0000 UTC m=+2587.304119240" watchObservedRunningTime="2026-07-07 03:01:02.547070038 +0000 UTC m=+2587.304619130" Jul 07 03:01:03.086868 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:03.086835 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:04.085607 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:04.085533 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:05.091725 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:05.091021 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:06.092459 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:06.092408 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:07.093639 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:07.093570 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:08.095087 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:08.095029 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:08.835209 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:08.835168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:01:09.137073 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:09.136950 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:09.583207 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:09.583173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:01:09.683820 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:09.683753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:01:10.177981 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:10.177933 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:11.219024 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:11.218972 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:01:11.663389 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:11.663347 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:35320->10.133.0.44:15021: read: connection reset by peer" Jul 07 03:01:12.085109 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.085066 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:12.578410 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.578381 2564 generic.go:356] "Generic (PLEG): container finished" podID="b755f034-0fb2-4303-a629-2cefd1c08947" containerID="c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5" exitCode=0 Jul 07 03:01:12.578853 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.578455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerDied","Data":"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5"} Jul 07 03:01:12.578948 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.578881 2564 scope.go:117] "RemoveContainer" containerID="c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5" Jul 07 03:01:12.580003 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.579970 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:12.580133 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.580058 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:12.580133 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.580108 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:12.580286 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.580238 2564 generic.go:356] "Generic (PLEG): container finished" podID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerID="769830b156d946e94b4c6a3235666aaf988c4a87aa931bfc2f1812531b4a8917" exitCode=0 Jul 07 03:01:12.580334 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.580311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" event={"ID":"7bd2d15b-598d-452b-8b55-38ea546e380d","Type":"ContainerDied","Data":"769830b156d946e94b4c6a3235666aaf988c4a87aa931bfc2f1812531b4a8917"} Jul 07 03:01:12.580679 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.580663 2564 scope.go:117] "RemoveContainer" containerID="769830b156d946e94b4c6a3235666aaf988c4a87aa931bfc2f1812531b4a8917" Jul 07 03:01:12.581553 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.581530 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:12.581615 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.581587 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:12.581652 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:12.581624 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:01:13.085490 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:13.085438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:13.591114 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:13.591069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerStarted","Data":"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364"} Jul 07 03:01:13.598587 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:13.598552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" event={"ID":"7bd2d15b-598d-452b-8b55-38ea546e380d","Type":"ContainerStarted","Data":"79023fff2d5c3f031bac5ac5862201ef8f717e2be71b43cce370d419dd54da24"} Jul 07 03:01:14.085560 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:14.085527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:14.085916 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:14.085886 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:14.109517 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:14.109484 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:14.111087 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:14.111049 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 03:01:15.085390 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:15.085345 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:15.109911 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:15.109875 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 03:01:16.085489 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:16.085384 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:16.110118 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:16.109722 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 03:01:16.512496 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:16.512384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:17.085461 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:17.085406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:17.086706 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:17.086673 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:17.109599 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:17.109562 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 03:01:18.085865 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:18.085814 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" podUID="7bd2d15b-598d-452b-8b55-38ea546e380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 07 03:01:18.109118 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:18.109067 2564 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 03:01:19.088352 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:19.088316 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:19.089289 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:19.089267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tzh7s" Jul 07 03:01:19.113284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:19.113265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:19.114240 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:19.114219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:19.184645 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:19.184613 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:01:20.646882 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:20.646840 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" containerID="cri-o://b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364" gracePeriod=30 Jul 07 03:01:25.893828 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.893805 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:25.967331 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967268 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967331 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967300 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967340 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967361 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mbq7\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967440 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967474 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967515 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967509 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967827 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967519 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:01:25.967827 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token\") pod \"b755f034-0fb2-4303-a629-2cefd1c08947\" (UID: \"b755f034-0fb2-4303-a629-2cefd1c08947\") " Jul 07 03:01:25.967827 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967655 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:01:25.967827 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967721 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data" (OuterVolumeSpecName: "istio-data") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:01:25.968037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967854 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 03:01:25.968037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967885 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-credential-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:25.968037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967907 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:25.968037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.967922 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-data\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:25.968214 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.968037 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:01:25.969633 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.969601 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:01:25.969753 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.969690 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token" (OuterVolumeSpecName: "istio-token") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:01:25.969969 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.969938 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 03:01:25.969969 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:25.969944 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7" (OuterVolumeSpecName: "kube-api-access-9mbq7") pod "b755f034-0fb2-4303-a629-2cefd1c08947" (UID: "b755f034-0fb2-4303-a629-2cefd1c08947"). InnerVolumeSpecName "kube-api-access-9mbq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:01:26.069189 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069170 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b755f034-0fb2-4303-a629-2cefd1c08947-istiod-ca-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.069189 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069190 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-istio-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.069324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069198 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b755f034-0fb2-4303-a629-2cefd1c08947-istio-podinfo\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.069324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069206 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbq7\" (UniqueName: \"kubernetes.io/projected/b755f034-0fb2-4303-a629-2cefd1c08947-kube-api-access-9mbq7\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.069324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069214 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-workload-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.069324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.069222 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b755f034-0fb2-4303-a629-2cefd1c08947-istio-envoy\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:01:26.675550 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.675524 2564 generic.go:356] "Generic (PLEG): container finished" podID="b755f034-0fb2-4303-a629-2cefd1c08947" containerID="b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364" exitCode=0 Jul 07 03:01:26.675746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.675629 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" Jul 07 03:01:26.675746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.675648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerDied","Data":"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364"} Jul 07 03:01:26.675746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.675687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq" event={"ID":"b755f034-0fb2-4303-a629-2cefd1c08947","Type":"ContainerDied","Data":"b3da8df9acb98025a2a39480aea9bbfa37219f4f65bd8f9e1b7d177019ba24aa"} Jul 07 03:01:26.675746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.675707 2564 scope.go:117] "RemoveContainer" containerID="b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364" Jul 07 03:01:26.684296 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.684281 2564 scope.go:117] "RemoveContainer" containerID="c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5" Jul 07 03:01:26.691509 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.691490 2564 scope.go:117] "RemoveContainer" containerID="b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364" Jul 07 03:01:26.691746 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:01:26.691729 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364\": container with ID starting with b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364 not found: ID does not exist" containerID="b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364" Jul 07 03:01:26.691857 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.691754 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364"} err="failed to get container status \"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364\": rpc error: code = NotFound desc = could not find container \"b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364\": container with ID starting with b38888cbf9963846469f8214d1ee4bf31d2caf18838f269e9be11e14a9cf4364 not found: ID does not exist" Jul 07 03:01:26.691857 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.691771 2564 scope.go:117] "RemoveContainer" containerID="c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5" Jul 07 03:01:26.691997 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:01:26.691981 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5\": container with ID starting with c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5 not found: ID does not exist" containerID="c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5" Jul 07 03:01:26.692033 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.692003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5"} err="failed to get container status \"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5\": rpc error: code = NotFound desc = could not find container \"c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5\": container with ID starting with c0356403b06ffe1b69b3c55604caedad0c31ebca232d3647185c060813371aa5 not found: ID does not exist" Jul 07 03:01:26.697502 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.697483 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:01:26.701079 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:26.701061 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gb7nq"] Jul 07 03:01:27.790410 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:27.790383 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" path="/var/lib/kubelet/pods/b755f034-0fb2-4303-a629-2cefd1c08947/volumes" Jul 07 03:01:30.097393 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:30.097358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:01:30.195479 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:30.195444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:01:56.899702 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:01:56.899665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:02:55.916308 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:02:55.916229 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:02:55.924824 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:02:55.924805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:03:12.795974 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:03:12.795937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:03:32.387081 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:03:32.387038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:03.591172 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:03.591131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:13.552029 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:13.551995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:13.623892 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:13.623862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:32.361434 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:32.361358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:34.988298 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:34.988205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:35.087840 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:35.087810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:04:35.854469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:04:35.854433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:05:10.256269 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:10.256208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:05:10.535061 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:10.535018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:05:17.849746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.849678 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerID="744f24a3f8fd8592de7875db472cfca133a9b72ef59648a3328bf1df43bb45e5" exitCode=137 Jul 07 03:05:17.849746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.849732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerDied","Data":"744f24a3f8fd8592de7875db472cfca133a9b72ef59648a3328bf1df43bb45e5"} Jul 07 03:05:17.850201 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.849802 2564 scope.go:117] "RemoveContainer" containerID="465c738ceed819707e0d56d2d8ab05bb55be2ddf857721d6ac5609aa34f2e709" Jul 07 03:05:17.850201 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.850130 2564 scope.go:117] "RemoveContainer" containerID="744f24a3f8fd8592de7875db472cfca133a9b72ef59648a3328bf1df43bb45e5" Jul 07 03:05:17.851132 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.851100 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:05:17.851212 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.851189 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:05:17.851310 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:17.851248 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:05:18.856448 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:18.856414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" event={"ID":"5f81fc67-aedb-4b88-867c-7804aadffb05","Type":"ContainerStarted","Data":"030bb149ae0eeb141f11e3f3292e79fb59eb2d22b8757f9a83911649784d78e5"} Jul 07 03:05:19.343423 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:19.343391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 03:05:19.343610 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:19.343586 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 03:05:19.343684 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:19.343644 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 03:05:20.343164 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:20.343126 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 03:05:20.343570 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:20.343185 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 03:05:21.343576 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:21.343540 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 03:05:21.343967 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:21.343592 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 03:05:22.343399 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:22.343361 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 03:05:22.343570 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:22.343416 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 03:05:22.570744 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:22.570707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:05:23.343376 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:23.343341 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 07 03:05:23.343543 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:23.343399 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 07 03:05:25.343138 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:25.343103 2564 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-c9ndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 03:05:25.343599 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:25.343166 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" podUID="5f81fc67-aedb-4b88-867c-7804aadffb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 03:05:25.892459 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:25.892421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 03:05:25.892876 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:25.892835 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 03:05:25.896506 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:25.896474 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-c9ndl" Jul 07 03:05:41.402757 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:05:41.402721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:07:03.490857 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:03.490820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:07:28.005512 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:28.005432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:07:28.234125 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:28.234092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:07:37.137883 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:37.137848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:07:55.947514 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:55.947487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:07:55.956482 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:07:55.956462 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:08:20.770807 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:08:20.770756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:08:20.837138 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:08:20.837105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:08:47.557498 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:08:47.557411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:08:47.789254 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:08:47.789215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:08:57.383334 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:08:57.383290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:09:43.580530 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:09:43.580494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:09:43.639522 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:09:43.639474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:10:09.331045 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:10:09.331012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:10:09.547337 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:10:09.547309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:10:21.540878 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:10:21.540775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:11:16.302505 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:11:16.302469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:11:54.442306 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:11:54.442216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:11:54.501040 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:11:54.500982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:14.379703 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.379666 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:14.380185 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380174 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.380233 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380191 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.380233 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380219 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.380233 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380229 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.380336 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380301 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.380336 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.380311 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b755f034-0fb2-4303-a629-2cefd1c08947" containerName="istio-proxy" Jul 07 03:12:14.383589 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.383565 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.387068 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.386553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-wlc7g\"" Jul 07 03:12:14.394503 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.394480 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:14.401313 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401450 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401450 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401572 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401572 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401572 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxh2\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401572 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401727 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.401727 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.401676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.421148 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.421122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:14.486450 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.486414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:14.502603 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.502745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.502745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.502745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxh2\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.502745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.502745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503037 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.502980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503200 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.503037 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503200 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.503075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503310 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.503246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503451 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.503431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.503652 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.503633 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.505393 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.505373 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.505512 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.505494 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.512404 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.512353 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.512404 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.512395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxh2\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2\") pod \"router-gateway-2-openshift-default-54c789bdc6-qlsdv\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.695705 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.695643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:14.841744 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.841717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:14.842627 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:12:14.842597 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a79d2a_9ba3_4840_980c_f53dc5ac000b.slice/crio-15f0289f60e8e83ec110855e967bc40bad013f487b050ed494947aa48a3aeb95 WatchSource:0}: Error finding container 15f0289f60e8e83ec110855e967bc40bad013f487b050ed494947aa48a3aeb95: Status 404 returned error can't find the container with id 15f0289f60e8e83ec110855e967bc40bad013f487b050ed494947aa48a3aeb95 Jul 07 03:12:14.844552 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.844522 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:12:14.844631 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.844593 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:12:14.844631 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.844625 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 03:12:14.933334 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.932128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" event={"ID":"e5a79d2a-9ba3-4840-980c-f53dc5ac000b","Type":"ContainerStarted","Data":"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0"} Jul 07 03:12:14.933334 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.932852 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" event={"ID":"e5a79d2a-9ba3-4840-980c-f53dc5ac000b","Type":"ContainerStarted","Data":"15f0289f60e8e83ec110855e967bc40bad013f487b050ed494947aa48a3aeb95"} Jul 07 03:12:14.971946 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:14.971840 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" podStartSLOduration=0.971823452 podStartE2EDuration="971.823452ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:14.969531291 +0000 UTC m=+3259.727080337" watchObservedRunningTime="2026-07-07 03:12:14.971823452 +0000 UTC m=+3259.729372476" Jul 07 03:12:15.696747 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:15.696708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:15.702553 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:15.702527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:15.937908 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:15.937875 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:15.938911 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:15.938894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:18.687378 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:18.687339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:23.468533 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:23.468417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:23.893025 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:23.892978 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:23.893255 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:23.893231 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" podUID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" containerName="istio-proxy" containerID="cri-o://c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0" gracePeriod=30 Jul 07 03:12:26.246150 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.246125 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:26.301743 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.301716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.301879 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.301764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.301879 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.301809 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.301975 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.301928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.301975 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.301966 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.302080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302004 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddxh2\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.302080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.302080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.302080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302003 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:12:26.302298 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302057 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:12:26.302298 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302133 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo\") pod \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\" (UID: \"e5a79d2a-9ba3-4840-980c-f53dc5ac000b\") " Jul 07 03:12:26.302298 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302152 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:12:26.302449 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302381 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.302449 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302398 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-data\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.302449 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302409 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-credential-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.302449 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302405 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:12:26.302449 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.302427 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 03:12:26.304230 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.304195 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:12:26.304340 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.304222 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 03:12:26.304340 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.304277 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:12:26.304340 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.304288 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2" (OuterVolumeSpecName: "kube-api-access-ddxh2") pod "e5a79d2a-9ba3-4840-980c-f53dc5ac000b" (UID: "e5a79d2a-9ba3-4840-980c-f53dc5ac000b"). InnerVolumeSpecName "kube-api-access-ddxh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:12:26.403519 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403495 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-token\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.403519 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403519 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-envoy\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.403669 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403528 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxh2\" (UniqueName: \"kubernetes.io/projected/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-kube-api-access-ddxh2\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.403669 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403537 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istiod-ca-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.403669 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403545 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-workload-socket\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.403669 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.403554 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5a79d2a-9ba3-4840-980c-f53dc5ac000b-istio-podinfo\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:12:26.998492 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.998459 2564 generic.go:356] "Generic (PLEG): container finished" podID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" containerID="c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0" exitCode=0 Jul 07 03:12:26.998676 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.998519 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" Jul 07 03:12:26.998676 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.998541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" event={"ID":"e5a79d2a-9ba3-4840-980c-f53dc5ac000b","Type":"ContainerDied","Data":"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0"} Jul 07 03:12:26.998676 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.998596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv" event={"ID":"e5a79d2a-9ba3-4840-980c-f53dc5ac000b","Type":"ContainerDied","Data":"15f0289f60e8e83ec110855e967bc40bad013f487b050ed494947aa48a3aeb95"} Jul 07 03:12:26.998676 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:26.998618 2564 scope.go:117] "RemoveContainer" containerID="c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0" Jul 07 03:12:27.009174 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:27.009151 2564 scope.go:117] "RemoveContainer" containerID="c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0" Jul 07 03:12:27.009944 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:12:27.009500 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0\": container with ID starting with c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0 not found: ID does not exist" containerID="c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0" Jul 07 03:12:27.009944 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:27.009551 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0"} err="failed to get container status \"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0\": rpc error: code = NotFound desc = could not find container \"c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0\": container with ID starting with c0ec22f962573d03818ca943f1979b19fe6b10876f161a0ff0c83a53382bd3a0 not found: ID does not exist" Jul 07 03:12:27.026671 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:27.026646 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:27.032139 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:27.032118 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-qlsdv"] Jul 07 03:12:27.792551 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:27.792521 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" path="/var/lib/kubelet/pods/e5a79d2a-9ba3-4840-980c-f53dc5ac000b/volumes" Jul 07 03:12:29.571270 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:29.571234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:30.163831 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:30.163771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:30.258902 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:30.258870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:38.326389 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:38.326356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:52.988116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:52.988073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:53.051064 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:53.051008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:12:55.979289 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:55.979262 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:12:55.989365 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:12:55.989344 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:13:13.500205 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:13:13.500162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:13:13.555156 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:13:13.555119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:03.840269 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.840228 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:14:03.840817 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.840707 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" containerName="istio-proxy" Jul 07 03:14:03.840817 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.840726 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" containerName="istio-proxy" Jul 07 03:14:03.840952 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.840838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a79d2a-9ba3-4840-980c-f53dc5ac000b" containerName="istio-proxy" Jul 07 03:14:03.843901 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.843879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.847619 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.847592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\"" Jul 07 03:14:03.847730 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.847624 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:14:03.853321 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.853299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:14:03.878609 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.878575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:03.994765 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.994958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.994958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mpzx\" (UniqueName: \"kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.994958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.994958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:03.994958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:03.994938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096347 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096347 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mpzx\" (UniqueName: \"kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096347 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096569 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096569 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096569 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096726 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096705 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096814 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096724 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.096866 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.096845 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.098870 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.098844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.099138 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.099118 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.104816 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.104773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mpzx\" (UniqueName: \"kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx\") pod \"rolling-upgrade-test-kserve-54bc76f55b-vqrcx\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.155410 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.155367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:04.170296 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.170249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:04.298591 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.298563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:14:04.300815 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:14:04.300753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57246009_81cd_4f17_af47_1fb18758b2ae.slice/crio-929f558de801256288bc2c62dbdd0fa8cc6ca9bbd5d871fc236f1f13f33bb086 WatchSource:0}: Error finding container 929f558de801256288bc2c62dbdd0fa8cc6ca9bbd5d871fc236f1f13f33bb086: Status 404 returned error can't find the container with id 929f558de801256288bc2c62dbdd0fa8cc6ca9bbd5d871fc236f1f13f33bb086 Jul 07 03:14:04.493850 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.493812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" event={"ID":"57246009-81cd-4f17-af47-1fb18758b2ae","Type":"ContainerStarted","Data":"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd"} Jul 07 03:14:04.494052 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.493881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" event={"ID":"57246009-81cd-4f17-af47-1fb18758b2ae","Type":"ContainerStarted","Data":"929f558de801256288bc2c62dbdd0fa8cc6ca9bbd5d871fc236f1f13f33bb086"} Jul 07 03:14:04.512080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:04.512022 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" podStartSLOduration=1.5120072979999999 podStartE2EDuration="1.512007298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:14:04.510532066 +0000 UTC m=+3369.268081109" watchObservedRunningTime="2026-07-07 03:14:04.512007298 +0000 UTC m=+3369.269556356" Jul 07 03:14:05.155958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:05.155919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:14.155565 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:14.155529 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:14.168003 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:14.167981 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:14.557685 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:14.557658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:14:37.255409 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:37.255373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:39.736764 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.736732 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:14:39.740319 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.740300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.750130 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.750097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:14:39.891922 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.891889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpkf\" (UniqueName: \"kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.892129 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.891941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.892129 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.891962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.892129 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.892028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.892129 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.892094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.892371 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.892244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993371 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993371 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993371 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993585 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993585 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993585 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpkf\" (UniqueName: \"kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993831 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993778 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993907 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993834 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.993907 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.993885 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.996075 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.996051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:39.996234 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:39.996219 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:40.002006 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.001984 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpkf\" (UniqueName: \"kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf\") pod \"rolling-upgrade-test-kserve-859df57bcb-q5g2s\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:40.051025 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.050998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:40.184493 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.184462 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:14:40.185690 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:14:40.185652 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac24fa2_c04e_48af_b506_2ff83709c44d.slice/crio-a481e19af3862d452691022106d26410602fa2607ac586f8cb0ff2507eaa8016 WatchSource:0}: Error finding container a481e19af3862d452691022106d26410602fa2607ac586f8cb0ff2507eaa8016: Status 404 returned error can't find the container with id a481e19af3862d452691022106d26410602fa2607ac586f8cb0ff2507eaa8016 Jul 07 03:14:40.682962 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.682925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" event={"ID":"7ac24fa2-c04e-48af-b506-2ff83709c44d","Type":"ContainerStarted","Data":"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e"} Jul 07 03:14:40.683160 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.682979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" event={"ID":"7ac24fa2-c04e-48af-b506-2ff83709c44d","Type":"ContainerStarted","Data":"a481e19af3862d452691022106d26410602fa2607ac586f8cb0ff2507eaa8016"} Jul 07 03:14:40.701757 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:40.701710 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" podStartSLOduration=1.701695457 podStartE2EDuration="1.701695457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:14:40.699528355 +0000 UTC m=+3405.457077380" watchObservedRunningTime="2026-07-07 03:14:40.701695457 +0000 UTC m=+3405.459244481" Jul 07 03:14:41.051744 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:41.051709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:50.051654 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.051565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:50.064693 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.064668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:50.743812 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.743764 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:14:50.749568 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.749466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:14:50.749568 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.749611 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:14:50.751308 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.751280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:50.812326 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:14:50.812286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:14:50.889357 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:50.889320 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:50.889551 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:50.889420 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:14:51.389397422 +0000 UTC m=+3416.146946428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:51.395339 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:51.395306 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:51.395708 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:51.395386 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:14:52.395364983 +0000 UTC m=+3417.152913988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:52.405268 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:52.405233 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:52.405650 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:52.405302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:14:54.405287616 +0000 UTC m=+3419.162836619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:54.423832 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:54.423798 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:54.424217 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:54.423874 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:14:58.423858624 +0000 UTC m=+3423.181407627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:58.456635 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:58.456602 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:14:58.457032 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:14:58.456669 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:15:06.456653762 +0000 UTC m=+3431.214202765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:15:05.770724 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:05.770661 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" podUID="57246009-81cd-4f17-af47-1fb18758b2ae" containerName="main" containerID="cri-o://1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd" gracePeriod=45 Jul 07 03:15:06.020695 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.020669 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:15:06.107256 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107270 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107368 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107469 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107452 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mpzx\" (UniqueName: \"kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx\") pod \"57246009-81cd-4f17-af47-1fb18758b2ae\" (UID: \"57246009-81cd-4f17-af47-1fb18758b2ae\") " Jul 07 03:15:06.107728 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107559 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:06.107728 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107635 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home" (OuterVolumeSpecName: "home") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:06.107728 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107706 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:06.107920 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107839 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.107920 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107858 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.107920 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.107873 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.109636 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.109610 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:15:06.109636 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.109627 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm" (OuterVolumeSpecName: "dshm") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:06.109798 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.109636 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx" (OuterVolumeSpecName: "kube-api-access-8mpzx") pod "57246009-81cd-4f17-af47-1fb18758b2ae" (UID: "57246009-81cd-4f17-af47-1fb18758b2ae"). InnerVolumeSpecName "kube-api-access-8mpzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:15:06.208796 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.208760 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mpzx\" (UniqueName: \"kubernetes.io/projected/57246009-81cd-4f17-af47-1fb18758b2ae-kube-api-access-8mpzx\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.208961 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.208808 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57246009-81cd-4f17-af47-1fb18758b2ae-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.208961 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.208818 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57246009-81cd-4f17-af47-1fb18758b2ae-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:06.512044 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:15:06.511950 2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:15:06.512044 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:15:06.512040 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs podName:7ac24fa2-c04e-48af-b506-2ff83709c44d nodeName:}" failed. No retries permitted until 2026-07-07 03:15:22.512015269 +0000 UTC m=+3447.269564271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-q5g2s" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found Jul 07 03:15:06.762384 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.762261 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" podUID="7ac24fa2-c04e-48af-b506-2ff83709c44d" containerName="main" containerID="cri-o://05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e" gracePeriod=45 Jul 07 03:15:06.816860 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.816824 2564 generic.go:356] "Generic (PLEG): container finished" podID="57246009-81cd-4f17-af47-1fb18758b2ae" containerID="1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd" exitCode=0 Jul 07 03:15:06.817282 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.816870 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" Jul 07 03:15:06.817282 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.816879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" event={"ID":"57246009-81cd-4f17-af47-1fb18758b2ae","Type":"ContainerDied","Data":"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd"} Jul 07 03:15:06.817282 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.816932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx" event={"ID":"57246009-81cd-4f17-af47-1fb18758b2ae","Type":"ContainerDied","Data":"929f558de801256288bc2c62dbdd0fa8cc6ca9bbd5d871fc236f1f13f33bb086"} Jul 07 03:15:06.817282 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.816963 2564 scope.go:117] "RemoveContainer" containerID="1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd" Jul 07 03:15:06.843347 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.843304 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:15:06.846644 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.846621 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-vqrcx"] Jul 07 03:15:06.873620 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.873601 2564 scope.go:117] "RemoveContainer" containerID="1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd" Jul 07 03:15:06.874009 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:15:06.873989 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd\": container with ID starting with 1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd not found: ID does not exist" containerID="1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd" Jul 07 03:15:06.874061 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:06.874021 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd"} err="failed to get container status \"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd\": rpc error: code = NotFound desc = could not find container \"1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd\": container with ID starting with 1ab4ed4e848ed7c9fbbc1a0ceaa3f4e43967f9af16f306b6968eea061b0affbd not found: ID does not exist" Jul 07 03:15:07.014253 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.014182 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:15:07.117472 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117525 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117558 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzpkf\" (UniqueName: \"kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf\") pod \"7ac24fa2-c04e-48af-b506-2ff83709c44d\" (UID: \"7ac24fa2-c04e-48af-b506-2ff83709c44d\") " Jul 07 03:15:07.117980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117755 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:07.117980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117881 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:07.117980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117895 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home" (OuterVolumeSpecName: "home") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:07.117980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.117968 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.119744 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.119724 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:15:07.119845 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.119823 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf" (OuterVolumeSpecName: "kube-api-access-jzpkf") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "kube-api-access-jzpkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:15:07.119845 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.119830 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm" (OuterVolumeSpecName: "dshm") pod "7ac24fa2-c04e-48af-b506-2ff83709c44d" (UID: "7ac24fa2-c04e-48af-b506-2ff83709c44d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:15:07.219005 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.218973 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.219005 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.218998 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac24fa2-c04e-48af-b506-2ff83709c44d-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.219005 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.219008 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.219227 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.219016 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac24fa2-c04e-48af-b506-2ff83709c44d-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.219227 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.219024 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzpkf\" (UniqueName: \"kubernetes.io/projected/7ac24fa2-c04e-48af-b506-2ff83709c44d-kube-api-access-jzpkf\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:07.790742 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.790706 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57246009-81cd-4f17-af47-1fb18758b2ae" path="/var/lib/kubelet/pods/57246009-81cd-4f17-af47-1fb18758b2ae/volumes" Jul 07 03:15:07.823708 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.823671 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ac24fa2-c04e-48af-b506-2ff83709c44d" containerID="05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e" exitCode=0 Jul 07 03:15:07.824183 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.823721 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" Jul 07 03:15:07.824183 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.823760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" event={"ID":"7ac24fa2-c04e-48af-b506-2ff83709c44d","Type":"ContainerDied","Data":"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e"} Jul 07 03:15:07.824183 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.823829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s" event={"ID":"7ac24fa2-c04e-48af-b506-2ff83709c44d","Type":"ContainerDied","Data":"a481e19af3862d452691022106d26410602fa2607ac586f8cb0ff2507eaa8016"} Jul 07 03:15:07.824183 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.823855 2564 scope.go:117] "RemoveContainer" containerID="05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e" Jul 07 03:15:07.832350 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.832328 2564 scope.go:117] "RemoveContainer" containerID="05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e" Jul 07 03:15:07.832623 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:15:07.832606 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e\": container with ID starting with 05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e not found: ID does not exist" containerID="05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e" Jul 07 03:15:07.832685 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.832631 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e"} err="failed to get container status \"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e\": rpc error: code = NotFound desc = could not find container \"05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e\": container with ID starting with 05b0b4c694aeaceea793a9e92eba947c2be98d713a15b3b7a0c62046abdd899e not found: ID does not exist" Jul 07 03:15:07.848890 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.848846 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:15:07.849050 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:07.848902 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-q5g2s"] Jul 07 03:15:09.790880 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:09.790846 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac24fa2-c04e-48af-b506-2ff83709c44d" path="/var/lib/kubelet/pods/7ac24fa2-c04e-48af-b506-2ff83709c44d/volumes" Jul 07 03:15:14.505439 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505408 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-765db479f9-wqkgm"] Jul 07 03:15:14.505883 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505768 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57246009-81cd-4f17-af47-1fb18758b2ae" containerName="main" Jul 07 03:15:14.505883 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505803 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="57246009-81cd-4f17-af47-1fb18758b2ae" containerName="main" Jul 07 03:15:14.505883 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505825 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac24fa2-c04e-48af-b506-2ff83709c44d" containerName="main" Jul 07 03:15:14.505883 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505835 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac24fa2-c04e-48af-b506-2ff83709c44d" containerName="main" Jul 07 03:15:14.506064 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505927 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="57246009-81cd-4f17-af47-1fb18758b2ae" containerName="main" Jul 07 03:15:14.506064 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.505944 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac24fa2-c04e-48af-b506-2ff83709c44d" containerName="main" Jul 07 03:15:14.510064 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.510048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.519090 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.519067 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-765db479f9-wqkgm"] Jul 07 03:15:14.573842 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.573814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gxd\" (UniqueName: \"kubernetes.io/projected/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-kube-api-access-w8gxd\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.573964 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.573851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-cert\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.674838 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.674812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gxd\" (UniqueName: \"kubernetes.io/projected/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-kube-api-access-w8gxd\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.674928 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.674855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-cert\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.677292 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.677274 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-cert\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.683922 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.683900 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gxd\" (UniqueName: \"kubernetes.io/projected/6b7c3df4-bbea-44c1-865e-1a1ef6b92b82-kube-api-access-w8gxd\") pod \"llmisvc-controller-manager-765db479f9-wqkgm\" (UID: \"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82\") " pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.819929 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.819906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:14.951464 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.951439 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-765db479f9-wqkgm"] Jul 07 03:15:14.953155 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:15:14.953128 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b7c3df4_bbea_44c1_865e_1a1ef6b92b82.slice/crio-9a414331805a14a970c05ba6b16905e86d225fc51384d97c7d8aef3bc703b887 WatchSource:0}: Error finding container 9a414331805a14a970c05ba6b16905e86d225fc51384d97c7d8aef3bc703b887: Status 404 returned error can't find the container with id 9a414331805a14a970c05ba6b16905e86d225fc51384d97c7d8aef3bc703b887 Jul 07 03:15:14.954512 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:14.954495 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 03:15:15.863845 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:15.863803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" event={"ID":"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82","Type":"ContainerStarted","Data":"263331c897886a5c6edc901ad647205e20974f0189d979d32c49acdd90709395"} Jul 07 03:15:15.864244 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:15.863857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" event={"ID":"6b7c3df4-bbea-44c1-865e-1a1ef6b92b82","Type":"ContainerStarted","Data":"9a414331805a14a970c05ba6b16905e86d225fc51384d97c7d8aef3bc703b887"} Jul 07 03:15:15.864244 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:15.863936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:15.881043 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:15.881002 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" podStartSLOduration=1.316622163 podStartE2EDuration="1.880988867s" totalImagesPullingTime="564.366704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:15:15.879596715 +0000 UTC m=+3440.637145753" watchObservedRunningTime="2026-07-07 03:15:15.880988867 +0000 UTC m=+3440.638537893" Jul 07 03:15:28.530951 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:28.530907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:15:28.595964 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:28.595934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:15:46.871189 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:46.871153 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-765db479f9-wqkgm" Jul 07 03:15:46.915844 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:46.915807 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 03:15:46.916091 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:46.916040 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" podUID="8faafaff-3a50-4922-b221-48162e703d4d" containerName="manager" containerID="cri-o://e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8" gracePeriod=30 Jul 07 03:15:47.167180 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.167159 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 03:15:47.326928 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.326895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") pod \"8faafaff-3a50-4922-b221-48162e703d4d\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " Jul 07 03:15:47.326928 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.326930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59pzp\" (UniqueName: \"kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp\") pod \"8faafaff-3a50-4922-b221-48162e703d4d\" (UID: \"8faafaff-3a50-4922-b221-48162e703d4d\") " Jul 07 03:15:47.329189 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.329158 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp" (OuterVolumeSpecName: "kube-api-access-59pzp") pod "8faafaff-3a50-4922-b221-48162e703d4d" (UID: "8faafaff-3a50-4922-b221-48162e703d4d"). InnerVolumeSpecName "kube-api-access-59pzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:15:47.329308 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.329200 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert" (OuterVolumeSpecName: "cert") pod "8faafaff-3a50-4922-b221-48162e703d4d" (UID: "8faafaff-3a50-4922-b221-48162e703d4d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:15:47.428448 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.428389 2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8faafaff-3a50-4922-b221-48162e703d4d-cert\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:47.428448 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:47.428409 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pzp\" (UniqueName: \"kubernetes.io/projected/8faafaff-3a50-4922-b221-48162e703d4d-kube-api-access-59pzp\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:15:48.016919 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.016878 2564 generic.go:356] "Generic (PLEG): container finished" podID="8faafaff-3a50-4922-b221-48162e703d4d" containerID="e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8" exitCode=0 Jul 07 03:15:48.017284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.016933 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" Jul 07 03:15:48.017284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.016935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" event={"ID":"8faafaff-3a50-4922-b221-48162e703d4d","Type":"ContainerDied","Data":"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8"} Jul 07 03:15:48.017284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.016991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2" event={"ID":"8faafaff-3a50-4922-b221-48162e703d4d","Type":"ContainerDied","Data":"be1c4654cfc6308cd1e6fb3b57098661802e181716949d21c0e326312c7d26e3"} Jul 07 03:15:48.017284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.017023 2564 scope.go:117] "RemoveContainer" containerID="e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8" Jul 07 03:15:48.027858 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.027836 2564 scope.go:117] "RemoveContainer" containerID="e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8" Jul 07 03:15:48.028147 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:15:48.028122 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8\": container with ID starting with e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8 not found: ID does not exist" containerID="e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8" Jul 07 03:15:48.028221 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.028155 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8"} err="failed to get container status \"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8\": rpc error: code = NotFound desc = could not find container \"e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8\": container with ID starting with e9e8b7a748aae59da2bdc816594d7c16fcdc64de875c10cae970e55b25d6f4c8 not found: ID does not exist" Jul 07 03:15:48.033858 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.033832 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 03:15:48.037601 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:48.037577 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-7fb88b8c57-q5hr2"] Jul 07 03:15:49.791769 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:15:49.791724 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8faafaff-3a50-4922-b221-48162e703d4d" path="/var/lib/kubelet/pods/8faafaff-3a50-4922-b221-48162e703d4d/volumes" Jul 07 03:16:18.744740 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:18.744655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:16:41.295231 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:41.295184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:16:53.833549 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.833515 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:16:53.834142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.834013 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8faafaff-3a50-4922-b221-48162e703d4d" containerName="manager" Jul 07 03:16:53.834142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.834033 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8faafaff-3a50-4922-b221-48162e703d4d" containerName="manager" Jul 07 03:16:53.834142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.834141 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8faafaff-3a50-4922-b221-48162e703d4d" containerName="manager" Jul 07 03:16:53.837259 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.837241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.840892 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.840870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:16:53.841007 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.840911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\"" Jul 07 03:16:53.845555 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.845529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:16:53.869316 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.869287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:16:53.907460 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.907570 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.907570 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.907660 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.907745 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:53.907817 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:53.907797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zff9x\" (UniqueName: \"kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.008882 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.008850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009047 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.008905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009047 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.008933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zff9x\" (UniqueName: \"kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009047 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.008955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009215 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.009098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009215 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.009163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009323 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.009249 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009421 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.009401 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.009503 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.009484 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.011272 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.011251 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.011377 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.011366 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.018924 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.018906 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zff9x\" (UniqueName: \"kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.149353 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.149253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:16:54.187261 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.187225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:16:54.290715 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.290687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:16:54.357122 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:54.357096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" event={"ID":"288f785a-9897-4ac9-987e-66d80eae0e24","Type":"ContainerStarted","Data":"feebc866dc800eecca7a8cfda63aca73945566d3add9e2059b1b7c8d452e1aa9"} Jul 07 03:16:55.363377 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:55.363332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" event={"ID":"288f785a-9897-4ac9-987e-66d80eae0e24","Type":"ContainerStarted","Data":"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197"} Jul 07 03:16:55.383019 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:55.382963 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" podStartSLOduration=2.382948249 podStartE2EDuration="2.382948249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:16:55.380794561 +0000 UTC m=+3540.138343598" watchObservedRunningTime="2026-07-07 03:16:55.382948249 +0000 UTC m=+3540.140497274" Jul 07 03:16:56.150142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:16:56.150110 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:01.270033 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:01.269999 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:17:04.150040 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:04.150006 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:04.162006 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:04.161985 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:04.426122 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:04.426040 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:26.778196 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:26.778161 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:17:26.788118 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:26.788081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:17:26.887846 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:26.887815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:17:41.808360 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:41.808297 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" podUID="288f785a-9897-4ac9-987e-66d80eae0e24" containerName="main" containerID="cri-o://c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197" gracePeriod=45 Jul 07 03:17:42.050117 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.050092 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:42.182714 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.182714 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.182714 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182704 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.182933 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182740 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zff9x\" (UniqueName: \"kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.182933 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.182933 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache\") pod \"288f785a-9897-4ac9-987e-66d80eae0e24\" (UID: \"288f785a-9897-4ac9-987e-66d80eae0e24\") " Jul 07 03:17:42.183104 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.182939 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home" (OuterVolumeSpecName: "home") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:17:42.183104 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.183042 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:17:42.183104 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.183084 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.183235 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.183154 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache" (OuterVolumeSpecName: "model-cache") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:17:42.184843 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.184820 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:17:42.184951 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.184937 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm" (OuterVolumeSpecName: "dshm") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:17:42.185054 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.185035 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x" (OuterVolumeSpecName: "kube-api-access-zff9x") pod "288f785a-9897-4ac9-987e-66d80eae0e24" (UID: "288f785a-9897-4ac9-987e-66d80eae0e24"). InnerVolumeSpecName "kube-api-access-zff9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:17:42.284249 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.284225 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288f785a-9897-4ac9-987e-66d80eae0e24-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.284249 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.284246 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.284372 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.284256 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zff9x\" (UniqueName: \"kubernetes.io/projected/288f785a-9897-4ac9-987e-66d80eae0e24-kube-api-access-zff9x\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.284372 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.284267 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.284372 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.284277 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288f785a-9897-4ac9-987e-66d80eae0e24-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:17:42.607940 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.607913 2564 generic.go:356] "Generic (PLEG): container finished" podID="288f785a-9897-4ac9-987e-66d80eae0e24" containerID="c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197" exitCode=0 Jul 07 03:17:42.608093 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.607969 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" Jul 07 03:17:42.608093 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.608021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" event={"ID":"288f785a-9897-4ac9-987e-66d80eae0e24","Type":"ContainerDied","Data":"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197"} Jul 07 03:17:42.608093 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.608089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6" event={"ID":"288f785a-9897-4ac9-987e-66d80eae0e24","Type":"ContainerDied","Data":"feebc866dc800eecca7a8cfda63aca73945566d3add9e2059b1b7c8d452e1aa9"} Jul 07 03:17:42.608227 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.608114 2564 scope.go:117] "RemoveContainer" containerID="c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197" Jul 07 03:17:42.624934 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.624906 2564 scope.go:117] "RemoveContainer" containerID="c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197" Jul 07 03:17:42.625269 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:17:42.625245 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197\": container with ID starting with c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197 not found: ID does not exist" containerID="c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197" Jul 07 03:17:42.625353 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.625276 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197"} err="failed to get container status \"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197\": rpc error: code = NotFound desc = could not find container \"c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197\": container with ID starting with c9deea561eef4ab48ccd57e779059828f8fbe542138ae7d3012a5b40505cb197 not found: ID does not exist" Jul 07 03:17:42.634644 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.634610 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:17:42.638649 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:42.638624 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-jjwd6"] Jul 07 03:17:43.791014 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:43.790976 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="288f785a-9897-4ac9-987e-66d80eae0e24" path="/var/lib/kubelet/pods/288f785a-9897-4ac9-987e-66d80eae0e24/volumes" Jul 07 03:17:49.444031 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.443945 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:17:49.444393 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.444307 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288f785a-9897-4ac9-987e-66d80eae0e24" containerName="main" Jul 07 03:17:49.444393 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.444321 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="288f785a-9897-4ac9-987e-66d80eae0e24" containerName="main" Jul 07 03:17:49.444393 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.444386 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="288f785a-9897-4ac9-987e-66d80eae0e24" containerName="main" Jul 07 03:17:49.448431 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.448411 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.452438 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.452418 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Jul 07 03:17:49.452558 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.452445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:17:49.455848 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.455822 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:17:49.487592 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.487568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:17:49.540927 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.540890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.541087 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.540953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bp2\" (UniqueName: \"kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.541152 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.541129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.541227 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.541209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.541281 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.541271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.541394 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.541371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.641961 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.641925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.641961 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.641966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.641985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bp2\" (UniqueName: \"kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642142 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642380 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642447 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642425 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.642505 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.642479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.644434 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.644405 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.644692 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.644677 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.651120 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.651098 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bp2\" (UniqueName: \"kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.759908 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.759851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:49.904287 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.904265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:17:49.906481 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:17:49.906453 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900cca4_0f0e_4eb6_ba72_75e72a55c426.slice/crio-00afd6a58317cd8a074dc755a8cfd82fb13887b972e9bb56149fb6fc2effdbd4 WatchSource:0}: Error finding container 00afd6a58317cd8a074dc755a8cfd82fb13887b972e9bb56149fb6fc2effdbd4: Status 404 returned error can't find the container with id 00afd6a58317cd8a074dc755a8cfd82fb13887b972e9bb56149fb6fc2effdbd4 Jul 07 03:17:49.909384 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:49.909354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:17:50.653431 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:50.653396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" event={"ID":"d900cca4-0f0e-4eb6-ba72-75e72a55c426","Type":"ContainerStarted","Data":"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075"} Jul 07 03:17:50.653928 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:50.653445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" event={"ID":"d900cca4-0f0e-4eb6-ba72-75e72a55c426","Type":"ContainerStarted","Data":"00afd6a58317cd8a074dc755a8cfd82fb13887b972e9bb56149fb6fc2effdbd4"} Jul 07 03:17:50.671632 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:50.671583 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" podStartSLOduration=1.671569103 podStartE2EDuration="1.671569103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:17:50.669451252 +0000 UTC m=+3595.427000295" watchObservedRunningTime="2026-07-07 03:17:50.671569103 +0000 UTC m=+3595.429118128" Jul 07 03:17:50.760103 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:50.760080 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:56.010742 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:56.010719 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:17:56.021378 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:56.021357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:17:59.760819 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:59.760765 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:17:59.773230 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:17:59.773208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:18:00.722613 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:00.722584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:18:02.215234 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:02.215190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:14.549636 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:14.549575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:15.215155 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:15.215122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:23.126272 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:23.126235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:23.996120 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:23.996089 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:18:24.004985 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:24.004958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:24.099634 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:24.099584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:39.018274 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.018227 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" podUID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" containerName="main" containerID="cri-o://fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075" gracePeriod=45 Jul 07 03:18:39.261912 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.261885 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:18:39.323008 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.322966 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323008 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323258 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323071 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323258 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bp2\" (UniqueName: \"kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323258 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323258 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home\") pod \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\" (UID: \"d900cca4-0f0e-4eb6-ba72-75e72a55c426\") " Jul 07 03:18:39.323478 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323317 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache" (OuterVolumeSpecName: "model-cache") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:18:39.323478 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323360 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:18:39.323478 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323464 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.323478 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323479 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.323718 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.323686 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home" (OuterVolumeSpecName: "home") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:18:39.325341 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.325312 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm" (OuterVolumeSpecName: "dshm") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:18:39.325341 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.325316 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:18:39.325496 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.325396 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2" (OuterVolumeSpecName: "kube-api-access-72bp2") pod "d900cca4-0f0e-4eb6-ba72-75e72a55c426" (UID: "d900cca4-0f0e-4eb6-ba72-75e72a55c426"). InnerVolumeSpecName "kube-api-access-72bp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:18:39.424209 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.424147 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bp2\" (UniqueName: \"kubernetes.io/projected/d900cca4-0f0e-4eb6-ba72-75e72a55c426-kube-api-access-72bp2\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.424209 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.424201 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d900cca4-0f0e-4eb6-ba72-75e72a55c426-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.424209 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.424213 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.424209 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.424224 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d900cca4-0f0e-4eb6-ba72-75e72a55c426-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:18:39.905235 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.905203 2564 generic.go:356] "Generic (PLEG): container finished" podID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" containerID="fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075" exitCode=0 Jul 07 03:18:39.905425 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.905248 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" Jul 07 03:18:39.905425 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.905293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" event={"ID":"d900cca4-0f0e-4eb6-ba72-75e72a55c426","Type":"ContainerDied","Data":"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075"} Jul 07 03:18:39.905425 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.905347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc" event={"ID":"d900cca4-0f0e-4eb6-ba72-75e72a55c426","Type":"ContainerDied","Data":"00afd6a58317cd8a074dc755a8cfd82fb13887b972e9bb56149fb6fc2effdbd4"} Jul 07 03:18:39.905425 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.905369 2564 scope.go:117] "RemoveContainer" containerID="fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075" Jul 07 03:18:39.914615 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.914593 2564 scope.go:117] "RemoveContainer" containerID="fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075" Jul 07 03:18:39.914949 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:18:39.914930 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075\": container with ID starting with fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075 not found: ID does not exist" containerID="fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075" Jul 07 03:18:39.915008 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.914960 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075"} err="failed to get container status \"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075\": rpc error: code = NotFound desc = could not find container \"fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075\": container with ID starting with fb0475e3b0c5786854fdbed696f65e87576d9aaa1d5158492945697351dcb075 not found: ID does not exist" Jul 07 03:18:39.924512 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.924482 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:18:39.930324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:39.930287 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-hcvbc"] Jul 07 03:18:41.791046 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:41.791008 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" path="/var/lib/kubelet/pods/d900cca4-0f0e-4eb6-ba72-75e72a55c426/volumes" Jul 07 03:18:57.979810 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:57.979727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:58.112810 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:58.112752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:58.188776 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:58.188736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:18:58.335224 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:18:58.335185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:24.515587 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.515501 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:19:24.516367 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.515982 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" containerName="main" Jul 07 03:19:24.516367 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.516002 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" containerName="main" Jul 07 03:19:24.516367 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.516100 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d900cca4-0f0e-4eb6-ba72-75e72a55c426" containerName="main" Jul 07 03:19:24.518998 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.518980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.523122 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.523103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\"" Jul 07 03:19:24.523231 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.523163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:19:24.531049 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.531031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:19:24.554129 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.554102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:24.620675 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.620773 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.620773 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8kjz\" (UniqueName: \"kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.620906 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.620906 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.620906 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.620832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726162 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726312 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726312 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726312 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726312 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.726566 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.726337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8kjz\" (UniqueName: \"kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.727043 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.727012 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.728464 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.728427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.728605 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.728482 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.730102 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.730071 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.731105 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.731081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.734252 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.734234 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8kjz\" (UniqueName: \"kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.829300 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.829279 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:24.886543 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.886084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:24.966445 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:24.966416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:19:24.967592 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:19:24.967563 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23cf9d2_34b6_41ba_a15f_4af6e6fd0733.slice/crio-54a248ba37b9f94dff86e4026961e09222f01ef22c81c8e03e012102a447fad3 WatchSource:0}: Error finding container 54a248ba37b9f94dff86e4026961e09222f01ef22c81c8e03e012102a447fad3: Status 404 returned error can't find the container with id 54a248ba37b9f94dff86e4026961e09222f01ef22c81c8e03e012102a447fad3 Jul 07 03:19:25.137840 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:25.137728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" event={"ID":"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733","Type":"ContainerStarted","Data":"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46"} Jul 07 03:19:25.137840 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:25.137820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" event={"ID":"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733","Type":"ContainerStarted","Data":"54a248ba37b9f94dff86e4026961e09222f01ef22c81c8e03e012102a447fad3"} Jul 07 03:19:25.154719 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:25.154675 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" podStartSLOduration=1.154659575 podStartE2EDuration="1.154659575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:19:25.153627698 +0000 UTC m=+3689.911176741" watchObservedRunningTime="2026-07-07 03:19:25.154659575 +0000 UTC m=+3689.912208602" Jul 07 03:19:25.829462 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:25.829426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:34.830049 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:34.830014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:34.844509 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:34.844476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:35.200897 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:35.200866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:19:45.739287 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:45.739254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:47.359048 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:47.359017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:52.923761 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:52.923723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:57.629629 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:57.629597 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:19:57.631523 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:57.631480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:19:57.690556 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:19:57.690511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:12.657852 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.657758 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" podUID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" containerName="main" containerID="cri-o://e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46" gracePeriod=45 Jul 07 03:20:12.903094 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.903072 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:20:12.933040 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.932975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933040 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933040 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933035 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933277 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933277 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933158 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8kjz\" (UniqueName: \"kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933277 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933185 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home\") pod \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\" (UID: \"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733\") " Jul 07 03:20:12.933444 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933318 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:20:12.933444 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933342 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache" (OuterVolumeSpecName: "model-cache") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:20:12.933547 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933449 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:12.933547 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933468 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:12.933547 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.933535 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home" (OuterVolumeSpecName: "home") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:20:12.935287 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.935258 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm" (OuterVolumeSpecName: "dshm") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:20:12.935388 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.935322 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:20:12.935657 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:12.935638 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz" (OuterVolumeSpecName: "kube-api-access-b8kjz") pod "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" (UID: "e23cf9d2-34b6-41ba-a15f-4af6e6fd0733"). InnerVolumeSpecName "kube-api-access-b8kjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:20:13.034478 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.034452 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8kjz\" (UniqueName: \"kubernetes.io/projected/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-kube-api-access-b8kjz\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:13.034580 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.034477 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:13.034580 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.034492 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:13.034580 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.034504 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:13.374018 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.373985 2564 generic.go:356] "Generic (PLEG): container finished" podID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" containerID="e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46" exitCode=0 Jul 07 03:20:13.374166 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.374048 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" Jul 07 03:20:13.374166 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.374077 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" event={"ID":"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733","Type":"ContainerDied","Data":"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46"} Jul 07 03:20:13.374166 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.374139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw" event={"ID":"e23cf9d2-34b6-41ba-a15f-4af6e6fd0733","Type":"ContainerDied","Data":"54a248ba37b9f94dff86e4026961e09222f01ef22c81c8e03e012102a447fad3"} Jul 07 03:20:13.374166 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.374165 2564 scope.go:117] "RemoveContainer" containerID="e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46" Jul 07 03:20:13.384248 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.384229 2564 scope.go:117] "RemoveContainer" containerID="e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46" Jul 07 03:20:13.384540 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:20:13.384518 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46\": container with ID starting with e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46 not found: ID does not exist" containerID="e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46" Jul 07 03:20:13.384623 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.384547 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46"} err="failed to get container status \"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46\": rpc error: code = NotFound desc = could not find container \"e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46\": container with ID starting with e5c8f9789d6bd3e3638dc5011771af163c6c0827c3b8deb78e8fc19504e6ce46 not found: ID does not exist" Jul 07 03:20:13.397292 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.397263 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:20:13.399834 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.399805 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-6w9zw"] Jul 07 03:20:13.791023 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:13.790982 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" path="/var/lib/kubelet/pods/e23cf9d2-34b6-41ba-a15f-4af6e6fd0733/volumes" Jul 07 03:20:24.483933 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.483896 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:20:24.484396 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.484377 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" containerName="main" Jul 07 03:20:24.484441 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.484401 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" containerName="main" Jul 07 03:20:24.484529 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.484517 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23cf9d2-34b6-41ba-a15f-4af6e6fd0733" containerName="main" Jul 07 03:20:24.489359 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.489337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.492098 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.492016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:20:24.493166 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.493148 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\"" Jul 07 03:20:24.497935 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.497914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:20:24.523553 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.523525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:24.528623 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2v5g\" (UniqueName: \"kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.528701 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.528701 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.528771 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.528771 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.528857 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.528777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.629949 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.629918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.629949 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.629952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630147 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.629974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630147 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630263 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v5g\" (UniqueName: \"kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630263 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630359 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630303 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630415 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630372 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.630474 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.630415 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.632635 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.632612 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.632825 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.632806 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.639522 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.639499 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v5g\" (UniqueName: \"kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.691570 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.691541 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:24.695685 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.695669 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.698156 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.698136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-bzglc\"" Jul 07 03:20:24.705333 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.705312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:24.722731 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.722704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:24.801314 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.801288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:24.831812 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.831760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.831931 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.831856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.831931 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.831885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xm5\" (UniqueName: \"kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.932710 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.932674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.932710 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.932714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.932922 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.932734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xm5\" (UniqueName: \"kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.933070 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.933050 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.935522 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.935499 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:24.936998 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.936978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:20:24.938986 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:20:24.938965 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c37c1b_fc18_45ab_9476_8192af70331e.slice/crio-727a779bd37eeab3e1cb9172baa46630be6c464a2b0fa40d65d9d975d0efcd9c WatchSource:0}: Error finding container 727a779bd37eeab3e1cb9172baa46630be6c464a2b0fa40d65d9d975d0efcd9c: Status 404 returned error can't find the container with id 727a779bd37eeab3e1cb9172baa46630be6c464a2b0fa40d65d9d975d0efcd9c Jul 07 03:20:24.941325 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:24.941301 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xm5\" (UniqueName: \"kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:25.006544 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.006522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:25.138092 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.138064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:25.139769 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:20:25.139744 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458a0f3d_4c8a_4346_ade5_1afb62205728.slice/crio-aa634bcf1879cfe3bf062bf7e11d61719e8753c1c7708e66b3254b14bb70bcd0 WatchSource:0}: Error finding container aa634bcf1879cfe3bf062bf7e11d61719e8753c1c7708e66b3254b14bb70bcd0: Status 404 returned error can't find the container with id aa634bcf1879cfe3bf062bf7e11d61719e8753c1c7708e66b3254b14bb70bcd0 Jul 07 03:20:25.141989 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.141970 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 03:20:25.435706 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.435490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" event={"ID":"458a0f3d-4c8a-4346-ade5-1afb62205728","Type":"ContainerStarted","Data":"aa634bcf1879cfe3bf062bf7e11d61719e8753c1c7708e66b3254b14bb70bcd0"} Jul 07 03:20:25.437025 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.437000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" event={"ID":"44c37c1b-fc18-45ab-9476-8192af70331e","Type":"ContainerStarted","Data":"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96"} Jul 07 03:20:25.437319 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.437045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" event={"ID":"44c37c1b-fc18-45ab-9476-8192af70331e","Type":"ContainerStarted","Data":"727a779bd37eeab3e1cb9172baa46630be6c464a2b0fa40d65d9d975d0efcd9c"} Jul 07 03:20:25.457581 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.457500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" podStartSLOduration=1.4574824290000001 podStartE2EDuration="1.457482429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:20:25.455567365 +0000 UTC m=+3750.213116413" watchObservedRunningTime="2026-07-07 03:20:25.457482429 +0000 UTC m=+3750.215031454" Jul 07 03:20:25.802371 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:25.802339 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:26.444554 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:26.444502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" event={"ID":"458a0f3d-4c8a-4346-ade5-1afb62205728","Type":"ContainerStarted","Data":"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d"} Jul 07 03:20:26.444835 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:26.444811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:26.461673 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:26.461627 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" podStartSLOduration=1.35629628 podStartE2EDuration="2.461609888s" totalImagesPullingTime="1.105313608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:20:26.460643765 +0000 UTC m=+3751.218192806" watchObservedRunningTime="2026-07-07 03:20:26.461609888 +0000 UTC m=+3751.219158912" Jul 07 03:20:32.774580 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:32.774543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:34.801844 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:34.801774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:34.815202 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:34.815170 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:35.503480 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:35.503454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:20:57.454777 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:57.454745 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:58.873312 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.869435 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:58.876216 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.876189 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:20:58.879404 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.879370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:58.888518 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:20:58.888479 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 07 03:20:58.888518 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:20:58.888518 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:20:58.888518 ip-10-0-137-109 kubenswrapper[2564]: > containerID="767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" cmd=["/bin/sleep","15"] Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.888537 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" message="" err=< Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: > Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:20:58.888548 2564 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: > pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerName="main" containerID="cri-o://767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" Jul 07 03:20:58.888832 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.888593 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerName="main" containerID="cri-o://767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" gracePeriod=60 Jul 07 03:20:58.940309 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:58.940274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:20:59.150645 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.150612 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:59.217949 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.217914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds\") pod \"458a0f3d-4c8a-4346-ade5-1afb62205728\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " Jul 07 03:20:59.218135 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.217956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57xm5\" (UniqueName: \"kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5\") pod \"458a0f3d-4c8a-4346-ade5-1afb62205728\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " Jul 07 03:20:59.218135 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.218007 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs\") pod \"458a0f3d-4c8a-4346-ade5-1afb62205728\" (UID: \"458a0f3d-4c8a-4346-ade5-1afb62205728\") " Jul 07 03:20:59.218246 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.218161 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "458a0f3d-4c8a-4346-ade5-1afb62205728" (UID: "458a0f3d-4c8a-4346-ade5-1afb62205728"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:20:59.218291 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.218273 2564 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/458a0f3d-4c8a-4346-ade5-1afb62205728-tokenizer-uds\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:59.220298 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.220268 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5" (OuterVolumeSpecName: "kube-api-access-57xm5") pod "458a0f3d-4c8a-4346-ade5-1afb62205728" (UID: "458a0f3d-4c8a-4346-ade5-1afb62205728"). InnerVolumeSpecName "kube-api-access-57xm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:20:59.220410 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.220309 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "458a0f3d-4c8a-4346-ade5-1afb62205728" (UID: "458a0f3d-4c8a-4346-ade5-1afb62205728"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:20:59.318602 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.318568 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57xm5\" (UniqueName: \"kubernetes.io/projected/458a0f3d-4c8a-4346-ade5-1afb62205728-kube-api-access-57xm5\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:59.318602 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.318595 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458a0f3d-4c8a-4346-ade5-1afb62205728-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:20:59.610891 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.610860 2564 generic.go:356] "Generic (PLEG): container finished" podID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerID="767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" exitCode=0 Jul 07 03:20:59.611096 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.610907 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" Jul 07 03:20:59.611096 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.610933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" event={"ID":"458a0f3d-4c8a-4346-ade5-1afb62205728","Type":"ContainerDied","Data":"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d"} Jul 07 03:20:59.611096 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.610973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t" event={"ID":"458a0f3d-4c8a-4346-ade5-1afb62205728","Type":"ContainerDied","Data":"aa634bcf1879cfe3bf062bf7e11d61719e8753c1c7708e66b3254b14bb70bcd0"} Jul 07 03:20:59.611096 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.610993 2564 scope.go:117] "RemoveContainer" containerID="767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" Jul 07 03:20:59.621154 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.621135 2564 scope.go:117] "RemoveContainer" containerID="767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" Jul 07 03:20:59.621412 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:20:59.621389 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d\": container with ID starting with 767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d not found: ID does not exist" containerID="767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d" Jul 07 03:20:59.621481 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.621419 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d"} err="failed to get container status \"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d\": rpc error: code = NotFound desc = could not find container \"767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d\": container with ID starting with 767a1f6bbb163d201586ef29cc5ceb50a1ddb1227eb00eb930e6247b01129b3d not found: ID does not exist" Jul 07 03:20:59.633268 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.633244 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:59.635096 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.635075 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59hpg5t"] Jul 07 03:20:59.791734 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:20:59.791697 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" path="/var/lib/kubelet/pods/458a0f3d-4c8a-4346-ade5-1afb62205728/volumes" Jul 07 03:21:13.903538 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:13.903486 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" podUID="44c37c1b-fc18-45ab-9476-8192af70331e" containerName="main" containerID="cri-o://8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96" gracePeriod=45 Jul 07 03:21:14.142076 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.142048 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:21:14.231685 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231624 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.231685 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.231870 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.231870 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231739 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.231870 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2v5g\" (UniqueName: \"kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.231870 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231838 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home\") pod \"44c37c1b-fc18-45ab-9476-8192af70331e\" (UID: \"44c37c1b-fc18-45ab-9476-8192af70331e\") " Jul 07 03:21:14.232060 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.231899 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache" (OuterVolumeSpecName: "model-cache") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:21:14.232114 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.232027 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:21:14.232181 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.232112 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home" (OuterVolumeSpecName: "home") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:21:14.232237 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.232176 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.232237 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.232194 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.233998 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.233957 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:21:14.233998 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.233978 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm" (OuterVolumeSpecName: "dshm") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:21:14.234143 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.234022 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g" (OuterVolumeSpecName: "kube-api-access-z2v5g") pod "44c37c1b-fc18-45ab-9476-8192af70331e" (UID: "44c37c1b-fc18-45ab-9476-8192af70331e"). InnerVolumeSpecName "kube-api-access-z2v5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:21:14.333544 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.333517 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.333544 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.333540 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c37c1b-fc18-45ab-9476-8192af70331e-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.333671 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.333552 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2v5g\" (UniqueName: \"kubernetes.io/projected/44c37c1b-fc18-45ab-9476-8192af70331e-kube-api-access-z2v5g\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.333671 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.333560 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c37c1b-fc18-45ab-9476-8192af70331e-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:21:14.688877 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.688847 2564 generic.go:356] "Generic (PLEG): container finished" podID="44c37c1b-fc18-45ab-9476-8192af70331e" containerID="8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96" exitCode=0 Jul 07 03:21:14.689055 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.688890 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" Jul 07 03:21:14.689055 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.688942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" event={"ID":"44c37c1b-fc18-45ab-9476-8192af70331e","Type":"ContainerDied","Data":"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96"} Jul 07 03:21:14.689055 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.688999 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht" event={"ID":"44c37c1b-fc18-45ab-9476-8192af70331e","Type":"ContainerDied","Data":"727a779bd37eeab3e1cb9172baa46630be6c464a2b0fa40d65d9d975d0efcd9c"} Jul 07 03:21:14.689055 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.689029 2564 scope.go:117] "RemoveContainer" containerID="8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96" Jul 07 03:21:14.698212 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.698196 2564 scope.go:117] "RemoveContainer" containerID="8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96" Jul 07 03:21:14.698463 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:21:14.698445 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96\": container with ID starting with 8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96 not found: ID does not exist" containerID="8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96" Jul 07 03:21:14.698510 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.698472 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96"} err="failed to get container status \"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96\": rpc error: code = NotFound desc = could not find container \"8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96\": container with ID starting with 8bdeabedfa2ad0a30dae6d33e8dddc8611cae492e6ee5859cd0685a1024dda96 not found: ID does not exist" Jul 07 03:21:14.711368 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.711348 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:21:14.713164 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:14.713143 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-cf8ht"] Jul 07 03:21:15.792194 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:15.792160 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c37c1b-fc18-45ab-9476-8192af70331e" path="/var/lib/kubelet/pods/44c37c1b-fc18-45ab-9476-8192af70331e/volumes" Jul 07 03:21:30.387400 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387368 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387700 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerName="main" Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387714 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerName="main" Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387740 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c37c1b-fc18-45ab-9476-8192af70331e" containerName="main" Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387745 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c37c1b-fc18-45ab-9476-8192af70331e" containerName="main" Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387813 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="458a0f3d-4c8a-4346-ade5-1afb62205728" containerName="main" Jul 07 03:21:30.387917 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.387830 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c37c1b-fc18-45ab-9476-8192af70331e" containerName="main" Jul 07 03:21:30.390869 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.390849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.395402 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.395387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:21:30.395482 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.395428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\"" Jul 07 03:21:30.416080 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.416054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:21:30.460290 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.460466 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.460466 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z662t\" (UniqueName: \"kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.460466 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.460600 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.460600 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.460520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.479370 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.479334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:30.561992 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.561955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z662t\" (UniqueName: \"kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562199 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562199 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562199 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562199 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562435 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562519 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562629 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562596 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.562738 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.562692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.564720 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.564701 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.565079 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.565062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.573984 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.573965 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z662t\" (UniqueName: \"kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.701272 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.701145 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:30.723006 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.722965 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:21:30.726321 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.726297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.729463 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.729435 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-5m9b7\"" Jul 07 03:21:30.736563 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.736535 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:21:30.764338 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.764122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjbs\" (UniqueName: \"kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.764338 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.764238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.764338 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.764287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.852120 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.852091 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:21:30.853609 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:21:30.853583 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc95baa2_d58a_4057_99d4_2f5340055d85.slice/crio-d874b08d7ad73656a0af813bb08f51c22b4594cb1183bb656f4c8a43933da317 WatchSource:0}: Error finding container d874b08d7ad73656a0af813bb08f51c22b4594cb1183bb656f4c8a43933da317: Status 404 returned error can't find the container with id d874b08d7ad73656a0af813bb08f51c22b4594cb1183bb656f4c8a43933da317 Jul 07 03:21:30.864825 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.864798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjbs\" (UniqueName: \"kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.864957 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.864889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.864957 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.864945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.865473 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.865448 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.867354 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.867335 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:30.873012 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:30.872991 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjbs\" (UniqueName: \"kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched4tk44\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:31.037447 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.037415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:31.170375 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.170350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:21:31.172155 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:21:31.172129 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1079394_fd34_4569_ba7f_950a8d80f568.slice/crio-d0773d7d87b1fc232adcf4565e4e534257fac0de4bf2b45d6c9fe0efafbf59a0 WatchSource:0}: Error finding container d0773d7d87b1fc232adcf4565e4e534257fac0de4bf2b45d6c9fe0efafbf59a0: Status 404 returned error can't find the container with id d0773d7d87b1fc232adcf4565e4e534257fac0de4bf2b45d6c9fe0efafbf59a0 Jul 07 03:21:31.592681 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.592647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:31.791684 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.791654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:21:31.791861 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.791697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" event={"ID":"dc95baa2-d58a-4057-99d4-2f5340055d85","Type":"ContainerStarted","Data":"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470"} Jul 07 03:21:31.791861 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.791719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" event={"ID":"dc95baa2-d58a-4057-99d4-2f5340055d85","Type":"ContainerStarted","Data":"d874b08d7ad73656a0af813bb08f51c22b4594cb1183bb656f4c8a43933da317"} Jul 07 03:21:31.791861 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.791735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" event={"ID":"e1079394-fd34-4569-ba7f-950a8d80f568","Type":"ContainerStarted","Data":"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6"} Jul 07 03:21:31.791861 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.791746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" event={"ID":"e1079394-fd34-4569-ba7f-950a8d80f568","Type":"ContainerStarted","Data":"d0773d7d87b1fc232adcf4565e4e534257fac0de4bf2b45d6c9fe0efafbf59a0"} Jul 07 03:21:31.806261 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.806215 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" podStartSLOduration=1.806200718 podStartE2EDuration="1.806200718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:21:31.803015405 +0000 UTC m=+3816.560564449" watchObservedRunningTime="2026-07-07 03:21:31.806200718 +0000 UTC m=+3816.563749742" Jul 07 03:21:31.819102 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:31.819053 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" podStartSLOduration=1.819032418 podStartE2EDuration="1.819032418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:21:31.818612497 +0000 UTC m=+3816.576161547" watchObservedRunningTime="2026-07-07 03:21:31.819032418 +0000 UTC m=+3816.576581444" Jul 07 03:21:32.701505 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:32.701471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:33.571174 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:33.571140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:33.670324 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:33.670287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:40.701991 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:40.701955 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:40.714705 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:40.714686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:40.845543 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:40.845518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:21:47.424167 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:47.424123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:58.059523 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:58.059482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:21:58.109081 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:21:58.109048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:02.795454 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:02.795427 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:22:04.368628 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.368590 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:22:04.370174 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.370142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:04.374714 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.374692 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:22:04.385486 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:22:04.385451 2564 log.go:32] "ExecSync cmd from runtime service failed" err=< Jul 07 03:22:04.385486 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:22:04.385486 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:22:04.385486 ip-10-0-137-109 kubenswrapper[2564]: > containerID="dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" cmd=["/bin/sleep","15"] Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.385500 2564 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" message="" err=< Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: > Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:22:04.385513 2564 kuberuntime_container.go:754] "PreStop hook failed" err=< Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: , stdout: , stderr: , exit code -1 Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" containerName="main" containerID="cri-o://dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" Jul 07 03:22:04.385835 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.385546 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" containerName="main" containerID="cri-o://dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" gracePeriod=60 Jul 07 03:22:04.466385 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.466344 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:04.634714 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.634691 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:22:04.754980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.754945 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs\") pod \"e1079394-fd34-4569-ba7f-950a8d80f568\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " Jul 07 03:22:04.755170 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.755057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds\") pod \"e1079394-fd34-4569-ba7f-950a8d80f568\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " Jul 07 03:22:04.755170 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.755079 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jjbs\" (UniqueName: \"kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs\") pod \"e1079394-fd34-4569-ba7f-950a8d80f568\" (UID: \"e1079394-fd34-4569-ba7f-950a8d80f568\") " Jul 07 03:22:04.755349 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.755324 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e1079394-fd34-4569-ba7f-950a8d80f568" (UID: "e1079394-fd34-4569-ba7f-950a8d80f568"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:22:04.757272 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.757240 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1079394-fd34-4569-ba7f-950a8d80f568" (UID: "e1079394-fd34-4569-ba7f-950a8d80f568"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:22:04.757272 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.757259 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs" (OuterVolumeSpecName: "kube-api-access-7jjbs") pod "e1079394-fd34-4569-ba7f-950a8d80f568" (UID: "e1079394-fd34-4569-ba7f-950a8d80f568"). InnerVolumeSpecName "kube-api-access-7jjbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:22:04.855990 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.855967 2564 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e1079394-fd34-4569-ba7f-950a8d80f568-tokenizer-uds\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:04.855990 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.855992 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjbs\" (UniqueName: \"kubernetes.io/projected/e1079394-fd34-4569-ba7f-950a8d80f568-kube-api-access-7jjbs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:04.856161 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.856005 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1079394-fd34-4569-ba7f-950a8d80f568-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:04.956849 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.956745 2564 generic.go:356] "Generic (PLEG): container finished" podID="e1079394-fd34-4569-ba7f-950a8d80f568" containerID="dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" exitCode=0 Jul 07 03:22:04.956849 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.956805 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" Jul 07 03:22:04.956849 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.956823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" event={"ID":"e1079394-fd34-4569-ba7f-950a8d80f568","Type":"ContainerDied","Data":"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6"} Jul 07 03:22:04.957095 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.956880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44" event={"ID":"e1079394-fd34-4569-ba7f-950a8d80f568","Type":"ContainerDied","Data":"d0773d7d87b1fc232adcf4565e4e534257fac0de4bf2b45d6c9fe0efafbf59a0"} Jul 07 03:22:04.957095 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.956908 2564 scope.go:117] "RemoveContainer" containerID="dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" Jul 07 03:22:04.966655 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.966634 2564 scope.go:117] "RemoveContainer" containerID="dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" Jul 07 03:22:04.966933 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:22:04.966913 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6\": container with ID starting with dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6 not found: ID does not exist" containerID="dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6" Jul 07 03:22:04.967011 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.966940 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6"} err="failed to get container status \"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6\": rpc error: code = NotFound desc = could not find container \"dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6\": container with ID starting with dad8e7a360f467c2a28fdfaf6822a6a9d9bf4622181d087d7dd31476bef0a0c6 not found: ID does not exist" Jul 07 03:22:04.980266 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.980241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:22:04.983400 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:04.983377 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched4tk44"] Jul 07 03:22:05.790847 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:05.790812 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" path="/var/lib/kubelet/pods/e1079394-fd34-4569-ba7f-950a8d80f568/volumes" Jul 07 03:22:14.356480 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:14.356451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:14.603627 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:14.603594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:19.392755 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.392660 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" podUID="dc95baa2-d58a-4057-99d4-2f5340055d85" containerName="main" containerID="cri-o://b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470" gracePeriod=45 Jul 07 03:22:19.646990 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.646930 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:22:19.665980 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.665946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666111 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.665986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666111 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666012 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666111 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z662t\" (UniqueName: \"kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666284 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666186 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache\") pod \"dc95baa2-d58a-4057-99d4-2f5340055d85\" (UID: \"dc95baa2-d58a-4057-99d4-2f5340055d85\") " Jul 07 03:22:19.666403 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666294 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home" (OuterVolumeSpecName: "home") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:22:19.666403 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666301 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:22:19.666506 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666432 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:19.666506 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666450 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:19.666506 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.666495 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:22:19.668362 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.668337 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm" (OuterVolumeSpecName: "dshm") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:22:19.668461 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.668441 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t" (OuterVolumeSpecName: "kube-api-access-z662t") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "kube-api-access-z662t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:22:19.668503 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.668484 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc95baa2-d58a-4057-99d4-2f5340055d85" (UID: "dc95baa2-d58a-4057-99d4-2f5340055d85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:22:19.767841 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.767817 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z662t\" (UniqueName: \"kubernetes.io/projected/dc95baa2-d58a-4057-99d4-2f5340055d85-kube-api-access-z662t\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:19.767841 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.767839 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:19.767973 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.767849 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc95baa2-d58a-4057-99d4-2f5340055d85-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:19.767973 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:19.767856 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc95baa2-d58a-4057-99d4-2f5340055d85-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:22:20.030231 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.030169 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc95baa2-d58a-4057-99d4-2f5340055d85" containerID="b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470" exitCode=0 Jul 07 03:22:20.030231 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.030214 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" Jul 07 03:22:20.030231 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.030224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" event={"ID":"dc95baa2-d58a-4057-99d4-2f5340055d85","Type":"ContainerDied","Data":"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470"} Jul 07 03:22:20.030398 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.030265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv" event={"ID":"dc95baa2-d58a-4057-99d4-2f5340055d85","Type":"ContainerDied","Data":"d874b08d7ad73656a0af813bb08f51c22b4594cb1183bb656f4c8a43933da317"} Jul 07 03:22:20.030398 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.030288 2564 scope.go:117] "RemoveContainer" containerID="b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470" Jul 07 03:22:20.039131 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.039115 2564 scope.go:117] "RemoveContainer" containerID="b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470" Jul 07 03:22:20.039373 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:22:20.039351 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470\": container with ID starting with b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470 not found: ID does not exist" containerID="b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470" Jul 07 03:22:20.039430 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.039380 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470"} err="failed to get container status \"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470\": rpc error: code = NotFound desc = could not find container \"b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470\": container with ID starting with b810b74554ee1f9c943d0b5e383816711de841273acfb3bcc422493811b8d470 not found: ID does not exist" Jul 07 03:22:20.047156 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.047136 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:22:20.050520 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:20.050500 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5c6bfv"] Jul 07 03:22:21.791702 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:21.791671 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc95baa2-d58a-4057-99d4-2f5340055d85" path="/var/lib/kubelet/pods/dc95baa2-d58a-4057-99d4-2f5340055d85/volumes" Jul 07 03:22:31.468947 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:31.468904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:53.435249 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435220 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435528 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc95baa2-d58a-4057-99d4-2f5340055d85" containerName="main" Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435538 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc95baa2-d58a-4057-99d4-2f5340055d85" containerName="main" Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435556 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" containerName="main" Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435561 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" containerName="main" Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435612 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1079394-fd34-4569-ba7f-950a8d80f568" containerName="main" Jul 07 03:22:53.435690 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.435623 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc95baa2-d58a-4057-99d4-2f5340055d85" containerName="main" Jul 07 03:22:53.438468 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.438450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.442232 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.442214 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\"" Jul 07 03:22:53.442323 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.442293 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Jul 07 03:22:53.447033 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.447008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:22:53.459588 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.459561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:53.538499 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.538499 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.538715 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.538715 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.538715 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rv8\" (UniqueName: \"kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.538715 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.538693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640133 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rv8\" (UniqueName: \"kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640343 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640343 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640343 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640343 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640343 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640719 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.640719 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.640707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.641052 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.641033 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.643079 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.643055 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.643211 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.643195 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.651467 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.651440 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rv8\" (UniqueName: \"kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.685609 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.685538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:22:53.749178 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.749149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:53.893274 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:53.893249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:22:53.894416 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:22:53.894386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2e9077_54dd_4008_a711_0429160e1175.slice/crio-7f9ae6c3a57d44d09e96a229856349e604d56b999080e2be484cd510c8215ab2 WatchSource:0}: Error finding container 7f9ae6c3a57d44d09e96a229856349e604d56b999080e2be484cd510c8215ab2: Status 404 returned error can't find the container with id 7f9ae6c3a57d44d09e96a229856349e604d56b999080e2be484cd510c8215ab2 Jul 07 03:22:54.197079 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:54.196989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" event={"ID":"fc2e9077-54dd-4008-a711-0429160e1175","Type":"ContainerStarted","Data":"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5"} Jul 07 03:22:54.197079 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:54.197040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" event={"ID":"fc2e9077-54dd-4008-a711-0429160e1175","Type":"ContainerStarted","Data":"7f9ae6c3a57d44d09e96a229856349e604d56b999080e2be484cd510c8215ab2"} Jul 07 03:22:54.216806 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:54.215311 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" podStartSLOduration=1.215291945 podStartE2EDuration="1.215291945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:22:54.213708192 +0000 UTC m=+3898.971257231" watchObservedRunningTime="2026-07-07 03:22:54.215291945 +0000 UTC m=+3898.972840971" Jul 07 03:22:54.750217 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:54.750186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:22:56.047526 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:56.047495 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:22:56.055986 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:22:56.055961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:23:03.749858 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:03.749814 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:23:03.762141 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:03.762115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:23:04.257425 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:04.257392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:23:11.921197 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:11.921151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:13.351453 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:13.351422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:13.408872 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:13.408844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:26.682871 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:26.682839 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:23:26.686254 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:26.686225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:26.780094 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:26.780063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:41.704721 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:41.704656 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" podUID="fc2e9077-54dd-4008-a711-0429160e1175" containerName="main" containerID="cri-o://4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5" gracePeriod=45 Jul 07 03:23:41.953157 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:41.953128 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:23:42.028116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028092 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028297 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028297 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5rv8\" (UniqueName: \"kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028297 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028297 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir\") pod \"fc2e9077-54dd-4008-a711-0429160e1175\" (UID: \"fc2e9077-54dd-4008-a711-0429160e1175\") " Jul 07 03:23:42.028502 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028321 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:23:42.028502 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028444 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home" (OuterVolumeSpecName: "home") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:23:42.028502 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028462 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-model-cache\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.028631 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.028573 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:23:42.030420 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.030391 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8" (OuterVolumeSpecName: "kube-api-access-m5rv8") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "kube-api-access-m5rv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 03:23:42.030420 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.030408 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm" (OuterVolumeSpecName: "dshm") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 03:23:42.030593 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.030430 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc2e9077-54dd-4008-a711-0429160e1175" (UID: "fc2e9077-54dd-4008-a711-0429160e1175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 03:23:42.129404 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.129383 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-home\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.129404 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.129403 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5rv8\" (UniqueName: \"kubernetes.io/projected/fc2e9077-54dd-4008-a711-0429160e1175-kube-api-access-m5rv8\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.129524 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.129413 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2e9077-54dd-4008-a711-0429160e1175-tls-certs\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.129524 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.129423 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-tmp-dir\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.129524 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.129431 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2e9077-54dd-4008-a711-0429160e1175-dshm\") on node \"ip-10-0-137-109.ec2.internal\" DevicePath \"\"" Jul 07 03:23:42.444002 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.443972 2564 generic.go:356] "Generic (PLEG): container finished" podID="fc2e9077-54dd-4008-a711-0429160e1175" containerID="4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5" exitCode=0 Jul 07 03:23:42.444132 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.444021 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" Jul 07 03:23:42.444132 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.444060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" event={"ID":"fc2e9077-54dd-4008-a711-0429160e1175","Type":"ContainerDied","Data":"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5"} Jul 07 03:23:42.444132 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.444112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n" event={"ID":"fc2e9077-54dd-4008-a711-0429160e1175","Type":"ContainerDied","Data":"7f9ae6c3a57d44d09e96a229856349e604d56b999080e2be484cd510c8215ab2"} Jul 07 03:23:42.444244 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.444138 2564 scope.go:117] "RemoveContainer" containerID="4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5" Jul 07 03:23:42.454315 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.454297 2564 scope.go:117] "RemoveContainer" containerID="4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5" Jul 07 03:23:42.454587 ip-10-0-137-109 kubenswrapper[2564]: E0707 03:23:42.454567 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5\": container with ID starting with 4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5 not found: ID does not exist" containerID="4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5" Jul 07 03:23:42.454633 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.454595 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5"} err="failed to get container status \"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5\": rpc error: code = NotFound desc = could not find container \"4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5\": container with ID starting with 4cd9b3100613fc07b7729432673f44b8422162090663afbfee0c03b1bdb24ac5 not found: ID does not exist" Jul 07 03:23:42.466189 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.466165 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:23:42.468877 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:42.468854 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4t4h8n"] Jul 07 03:23:43.790942 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:43.790883 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2e9077-54dd-4008-a711-0429160e1175" path="/var/lib/kubelet/pods/fc2e9077-54dd-4008-a711-0429160e1175/volumes" Jul 07 03:23:48.181305 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:48.181215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:23:48.242094 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:23:48.242062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:03.934116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:03.934061 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:03.934534 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:03.934214 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:05.341604 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:05.341575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:05.341997 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:05.341753 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:06.749338 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:06.749308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:06.749722 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:06.749408 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:08.162990 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:08.162966 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:08.163388 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:08.163088 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:09.558250 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:09.558220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:09.558674 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:09.558469 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:10.946775 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:10.946749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:10.947259 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:10.946991 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:12.377739 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:12.377705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:12.378207 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:12.377874 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:13.804241 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:13.804209 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:13.804669 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:13.804494 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:15.264599 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:15.264570 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:15.265155 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:15.264695 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:16.738019 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:16.737987 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:16.738564 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:16.738213 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:18.137809 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:18.137756 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:18.138450 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:18.138227 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:19.533354 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:19.533327 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:19.533747 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:19.533487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:20.911972 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:20.911936 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:20.912379 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:20.912103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:22.301638 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:22.301612 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:22.302066 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:22.301715 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:23.675971 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:23.675938 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:23.676372 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:23.675993 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:25.087392 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:25.087363 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/1.log" Jul 07 03:24:25.087756 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:25.087402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-tzh7s_7bd2d15b-598d-452b-8b55-38ea546e380d/istio-proxy/0.log" Jul 07 03:24:26.735578 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:26.735537 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ngnzl_fe608940-dc4c-464e-b0d7-6b8be3b52c99/discovery/0.log" Jul 07 03:24:26.755692 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:26.755664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/2.log" Jul 07 03:24:26.765401 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:26.765380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/1.log" Jul 07 03:24:27.923677 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:27.923647 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ngnzl_fe608940-dc4c-464e-b0d7-6b8be3b52c99/discovery/0.log" Jul 07 03:24:27.950084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:27.950060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/2.log" Jul 07 03:24:27.959214 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:27.959197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/1.log" Jul 07 03:24:29.053173 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:29.053145 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj_917bc873-ba60-4dd9-b14d-52eaac877e60/manager/0.log" Jul 07 03:24:29.062429 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:29.062410 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-z5gcf_2f0719cf-edcf-4f8f-b606-511d32185922/limitador/0.log" Jul 07 03:24:29.076943 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:29.076922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-4wdtd_8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa/manager/0.log" Jul 07 03:24:31.494605 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:31.494569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:31.649256 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:31.649216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:31.747007 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:31.746926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:34.484958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:34.484925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zj6zc_40706701-1bf7-40be-b011-fb80c37fe4c1/global-pull-secret-syncer/0.log" Jul 07 03:24:34.525290 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:34.525262 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" Jul 07 03:24:34.657746 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:34.657717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-109.ec2.internal_a8c8e4db5eb6f333e24623a8dc7ce358/haproxy/0.log" Jul 07 03:24:37.221624 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:37.221593 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:37.292380 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:37.292347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lc69h_27712a9b-7072-4450-85c5-abb283829105/konnectivity-agent/0.log" limit=1024 Jul 07 03:24:39.918887 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:39.918855 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-f5bhj_917bc873-ba60-4dd9-b14d-52eaac877e60/manager/0.log" Jul 07 03:24:39.940599 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:39.940575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-z5gcf_2f0719cf-edcf-4f8f-b606-511d32185922/limitador/0.log" Jul 07 03:24:39.965864 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:39.965829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6f9f468797-4wdtd_8c7ecf9b-60f5-4d8b-873c-731d5f70ebfa/manager/0.log" Jul 07 03:24:41.477220 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.477185 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dvq65_ccee1b2b-7739-40f1-98a8-7c5a11dc5833/kube-state-metrics/0.log" Jul 07 03:24:41.495262 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.495239 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dvq65_ccee1b2b-7739-40f1-98a8-7c5a11dc5833/kube-rbac-proxy-main/0.log" Jul 07 03:24:41.515327 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.515293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dvq65_ccee1b2b-7739-40f1-98a8-7c5a11dc5833/kube-rbac-proxy-self/0.log" Jul 07 03:24:41.566544 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.566522 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-k29c9_8c8ff000-61ee-43ae-917e-1b7511223832/monitoring-plugin/0.log" Jul 07 03:24:41.674707 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.674684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9qmc_30e2c724-5810-4d6e-bd70-3e1c519fd882/node-exporter/0.log" Jul 07 03:24:41.698766 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.698749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9qmc_30e2c724-5810-4d6e-bd70-3e1c519fd882/kube-rbac-proxy/0.log" Jul 07 03:24:41.719723 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:41.719706 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9qmc_30e2c724-5810-4d6e-bd70-3e1c519fd882/init-textfile/0.log" Jul 07 03:24:42.097338 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:42.097313 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vqw2m_6755d51e-edde-4e70-a628-513836117e82/prometheus-operator/0.log" Jul 07 03:24:42.120038 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:42.120017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vqw2m_6755d51e-edde-4e70-a628-513836117e82/kube-rbac-proxy/0.log" Jul 07 03:24:44.616643 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.616610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76476b5bdb-gh7mz_033074c9-4c42-4217-893d-d8cbebd392e4/console/0.log" Jul 07 03:24:44.937958 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.937767 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm"] Jul 07 03:24:44.938116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.938096 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2e9077-54dd-4008-a711-0429160e1175" containerName="main" Jul 07 03:24:44.938116 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.938107 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2e9077-54dd-4008-a711-0429160e1175" containerName="main" Jul 07 03:24:44.938242 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.938158 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2e9077-54dd-4008-a711-0429160e1175" containerName="main" Jul 07 03:24:44.941357 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.941335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:44.944151 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.944080 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"kube-root-ca.crt\"" Jul 07 03:24:44.944303 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.944286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"openshift-service-ca.crt\"" Jul 07 03:24:44.945661 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.945642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8t88\"/\"default-dockercfg-s74vw\"" Jul 07 03:24:44.952941 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:44.952915 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm"] Jul 07 03:24:45.024148 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.024109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-sys\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.024339 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.024179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-proc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.024339 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.024196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-podres\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.024339 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.024213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-lib-modules\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.024339 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.024231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zlc\" (UniqueName: \"kubernetes.io/projected/f15f1024-8b87-438a-8913-26aa55b4b919-kube-api-access-l4zlc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.124897 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-sys\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-proc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-podres\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-lib-modules\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124992 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-sys\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.124999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zlc\" (UniqueName: \"kubernetes.io/projected/f15f1024-8b87-438a-8913-26aa55b4b919-kube-api-access-l4zlc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125084 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.125079 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-proc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125333 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.125113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-podres\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.125333 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.125142 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f15f1024-8b87-438a-8913-26aa55b4b919-lib-modules\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.133083 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.133061 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zlc\" (UniqueName: \"kubernetes.io/projected/f15f1024-8b87-438a-8913-26aa55b4b919-kube-api-access-l4zlc\") pod \"perf-node-gather-daemonset-ss9nm\" (UID: \"f15f1024-8b87-438a-8913-26aa55b4b919\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.252010 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.251927 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.388574 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.388547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm"] Jul 07 03:24:45.390491 ip-10-0-137-109 kubenswrapper[2564]: W0707 03:24:45.390448 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf15f1024_8b87_438a_8913_26aa55b4b919.slice/crio-570f309e40ce70239820567a5b4ebd9c689d5ad6417077053f08018593f0d0f5 WatchSource:0}: Error finding container 570f309e40ce70239820567a5b4ebd9c689d5ad6417077053f08018593f0d0f5: Status 404 returned error can't find the container with id 570f309e40ce70239820567a5b4ebd9c689d5ad6417077053f08018593f0d0f5 Jul 07 03:24:45.757167 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.757132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" event={"ID":"f15f1024-8b87-438a-8913-26aa55b4b919","Type":"ContainerStarted","Data":"bfeed28824f430487ca30ef329d15df1999e1e0a24c82f6fff2fbed48e313e01"} Jul 07 03:24:45.757567 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.757183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" event={"ID":"f15f1024-8b87-438a-8913-26aa55b4b919","Type":"ContainerStarted","Data":"570f309e40ce70239820567a5b4ebd9c689d5ad6417077053f08018593f0d0f5"} Jul 07 03:24:45.757567 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.757212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:45.774213 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:45.774163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" podStartSLOduration=1.774147677 podStartE2EDuration="1.774147677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 03:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:24:45.771765824 +0000 UTC m=+4010.529314850" watchObservedRunningTime="2026-07-07 03:24:45.774147677 +0000 UTC m=+4010.531696743" Jul 07 03:24:46.198986 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:46.198954 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9g6n5_44d6268e-27d4-40eb-b265-ff0ddd3475d6/dns/0.log" Jul 07 03:24:46.219954 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:46.219930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9g6n5_44d6268e-27d4-40eb-b265-ff0ddd3475d6/kube-rbac-proxy/0.log" Jul 07 03:24:46.352728 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:46.352701 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq4gw_a946adcf-84da-4601-a697-821c2c425a08/dns-node-resolver/0.log" Jul 07 03:24:46.919963 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:46.919933 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j96jg_b434ae8a-907e-4260-aa20-16ba66f6e467/node-ca/0.log" Jul 07 03:24:48.069472 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:48.069438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ngnzl_fe608940-dc4c-464e-b0d7-6b8be3b52c99/discovery/0.log" Jul 07 03:24:48.111206 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:48.111171 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/2.log" Jul 07 03:24:48.119704 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:48.119684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/1.log" Jul 07 03:24:48.681056 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:48.681027 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrq6x_948fc827-33f2-4c0b-96d8-229300467070/serve-healthcheck-canary/0.log" Jul 07 03:24:49.155107 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:49.155079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzghs_b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e/kube-rbac-proxy/0.log" Jul 07 03:24:49.176678 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:49.176652 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzghs_b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e/exporter/0.log" Jul 07 03:24:49.199300 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:49.199280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzghs_b4e6b0b5-ad25-4afe-ad0e-431e3ba9475e/extractor/0.log" Jul 07 03:24:51.772903 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:51.772877 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-ss9nm" Jul 07 03:24:52.725131 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:52.725097 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-765db479f9-wqkgm_6b7c3df4-bbea-44c1-865e-1a1ef6b92b82/manager/0.log" Jul 07 03:24:56.083179 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:56.083149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ngnzl_fe608940-dc4c-464e-b0d7-6b8be3b52c99/discovery/0.log" Jul 07 03:24:56.104057 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:56.104024 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/2.log" Jul 07 03:24:56.112989 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:56.112961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/1.log" Jul 07 03:24:57.434713 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:57.434681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ngnzl_fe608940-dc4c-464e-b0d7-6b8be3b52c99/discovery/0.log" Jul 07 03:24:57.465606 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:57.465575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/2.log" Jul 07 03:24:57.474546 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:24:57.474524 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-c9ndl_5f81fc67-aedb-4b88-867c-7804aadffb05/istio-proxy/1.log" Jul 07 03:25:00.051245 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.051217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/kube-multus-additional-cni-plugins/0.log" Jul 07 03:25:00.070285 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.070267 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/egress-router-binary-copy/0.log" Jul 07 03:25:00.090448 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.090430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/cni-plugins/0.log" Jul 07 03:25:00.111095 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.111075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/bond-cni-plugin/0.log" Jul 07 03:25:00.130620 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.130601 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/routeoverride-cni/0.log" Jul 07 03:25:00.151948 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.151934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/whereabouts-cni-bincopy/0.log" Jul 07 03:25:00.170722 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.170704 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5zxmf_e3bc19ab-4f53-46d0-958e-52ff15ae512e/whereabouts-cni/0.log" Jul 07 03:25:00.571167 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.571148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z5h7d_b84a7f82-3bde-45ad-a2ff-7cfe221b1037/kube-multus/0.log" Jul 07 03:25:00.592622 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.592600 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t4vsh_b7b9e554-ea7a-4647-b208-0cbb59badeb6/network-metrics-daemon/0.log" Jul 07 03:25:00.612484 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:00.612468 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t4vsh_b7b9e554-ea7a-4647-b208-0cbb59badeb6/kube-rbac-proxy/0.log" Jul 07 03:25:01.679610 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.679585 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-controller/0.log" Jul 07 03:25:01.696193 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.696163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/1.log" Jul 07 03:25:01.696315 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.696199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovn-acl-logging/0.log" Jul 07 03:25:01.715204 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.715179 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/kube-rbac-proxy-node/0.log" Jul 07 03:25:01.736684 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.736660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/kube-rbac-proxy-ovn-metrics/0.log" Jul 07 03:25:01.757859 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.757835 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/northd/0.log" Jul 07 03:25:01.772587 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.772565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/nbdb/0.log" Jul 07 03:25:01.790957 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.790938 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/sbdb/0.log" Jul 07 03:25:01.840838 ip-10-0-137-109 kubenswrapper[2564]: I0707 03:25:01.840814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qz6ck_985a2683-98de-489c-9990-6960d54df436/ovnkube-controller/0.log"